all_learn_process/codes/chapter3/第三章学习.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第三章 PyTorch基础：Tensor和Autograd\n",
    "\n",
    "## 3.1 Tensor\n",
    "\n",
    "Tensor，又名张量，读者可能对这个名词似曾相识，因它不仅在PyTorch中出现过，它也是Theano、TensorFlow、\n",
    "Torch和MxNet中重要的数据结构。关于张量的本质不乏深度的剖析，但从工程角度来讲，可简单地认为它就是一个数组，且支持高效的科学计算。它可以是一个数（标量）、一维数组（向量）、二维数组（矩阵）和更高维的数组（高阶数据）。Tensor和Numpy的ndarrays类似，但PyTorch的tensor支持GPU加速。\n",
    "\n",
    "本节将系统讲解tensor的使用，力求面面俱到，但不会涉及每个函数。对于更多函数及其用法，读者可通过在IPython/Notebook中使用函数名加`?`查看帮助文档，或查阅PyTorch官方文档[^1]。\n",
    "\n",
    "[^1]: http://docs.pytorch.org"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T01:56:26.757713Z",
     "start_time": "2020-12-09T01:56:26.445530Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.2.0'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's begin\n",
    "import torch  as t\n",
    "t.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  3.1.1 基础操作\n",
    "\n",
    "学习过Numpy的读者会对本节内容感到非常熟悉，因tensor的接口有意设计成与Numpy类似，以方便用户使用。但不熟悉Numpy也没关系，本节内容并不要求先掌握Numpy。\n",
    "\n",
    "从接口的角度来讲，对tensor的操作可分为两类：\n",
    "\n",
    "1. `torch.function`，如`torch.save`等。\n",
    "2. 另一类是`tensor.function`，如`tensor.view`等。\n",
    "\n",
    "为方便使用，对tensor的大部分操作同时支持这两类接口，在本书中不做具体区分，如`torch.sum (torch.sum(a, b))`与`tensor.sum (a.sum(b))`功能等价。\n",
    "\n",
    "而从存储的角度来讲，对tensor的操作又可分为两类：\n",
    "\n",
    "1. 不会修改自身的数据，如 `a.add(b)`， 加法的结果会返回一个新的tensor。\n",
    "2. 会修改自身的数据，如 `a.add_(b)`， 加法的结果仍存储在a中，a被修改了。\n",
    "\n",
    "函数名以`_`结尾的都是inplace方式, 即会修改调用者自己的数据，在实际应用中需加以区分。\n",
    "\n",
    "#### 创建Tensor\n",
    "\n",
    "在PyTorch中新建tensor的方法有很多，具体如表3-1所示。\n",
    "\n",
    "表3-1: 常见新建tensor的方法\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|Tensor(\\*sizes)|基础构造函数|\n",
    "|tensor(data,)|类似np.array的构造函数|\n",
    "|ones(\\*sizes)|全1Tensor|\n",
    "|zeros(\\*sizes)|全0Tensor|\n",
    "|eye(\\*sizes)|对角线为1，其他为0|\n",
    "|arange(s,e,step|从s到e，步长为step|\n",
    "|linspace(s,e,steps)|从s到e，均匀切分成steps份|\n",
    "|rand/randn(\\*sizes)|均匀/标准分布|\n",
    "|normal(mean,std)/uniform(from,to)|正态分布/均匀分布|\n",
    "|randperm(m)|随机排列|\n",
    "\n",
    "这些创建方法都可以在创建的时候指定数据类型dtype和存放device(cpu/gpu).\n",
    "\n",
    "\n",
    "其中使用`Tensor`函数新建tensor是最复杂多变的方式，它既可以接收一个list，并根据list的数据新建tensor，也能根据指定的形状新建tensor，还能传入其他的tensor，下面举几个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T01:58:11.583326Z",
     "start_time": "2020-12-09T01:58:11.574365Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2.0681e+20, 2.6685e-09, 4.2315e+21],\n",
       "        [6.4464e-10, 2.0803e+23, 3.2691e+21]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 指定tensor的形状\n",
    "a = t.Tensor(2, 3)\n",
    "a # 数值取决于内存空间的状态，print时候可能overflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T01:58:22.346925Z",
     "start_time": "2020-12-09T01:58:22.341939Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 2., 3.],\n",
       "        [4., 5., 6.]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 用list的数据创建tensor\n",
    "b = t.Tensor([[1,2,3],[4,5,6]])\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T01:58:31.694730Z",
     "start_time": "2020-12-09T01:58:31.689744Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.tolist() # 把tensor转为list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tensor.size()`返回`torch.Size`对象，它是tuple的子类，但其使用方式与tuple略有区别"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T01:58:56.699301Z",
     "start_time": "2020-12-09T01:58:56.694314Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b_size = b.size()\n",
    "b_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T01:59:05.086120Z",
     "start_time": "2020-12-09T01:59:05.082102Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.numel() # b中元素总个数，2*3，等价于b.nelement()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T01:59:17.914963Z",
     "start_time": "2020-12-09T01:59:17.909946Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[4.4213e-38, 7.1326e-43, 4.4213e-38],\n",
       "         [7.1326e-43, 2.2981e-38, 7.1326e-43]]),\n",
       " tensor([2., 3.]))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 创建一个和b形状一样的tensor\n",
    "c = t.Tensor(b_size)\n",
    "# 创建一个元素为2和3的tensor\n",
    "d = t.Tensor((2, 3))\n",
    "c, d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "除了`tensor.size()`，还可以利用`tensor.shape`直接查看tensor的形状，`tensor.shape`等价于`tensor.size()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T01:59:47.195037Z",
     "start_time": "2020-12-09T01:59:47.190050Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "需要注意的是，`t.Tensor(*sizes)`创建tensor时，系统不会马上分配空间，只是会计算剩余的内存是否足够使用，使用到tensor时才会分配，而其它操作都是在创建完tensor之后马上进行空间分配。其它常用的创建tensor的方法举例如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:00:38.699732Z",
     "start_time": "2020-12-09T02:00:38.693748Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.ones(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:00:45.073997Z",
     "start_time": "2020-12-09T02:00:45.066018Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.],\n",
       "        [0., 0., 0.]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.zeros(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:01:27.131679Z",
     "start_time": "2020-12-09T02:01:27.127690Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1, 2, 3, 4, 5])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.arange(1, 6, 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:00:59.182705Z",
     "start_time": "2020-12-09T02:00:59.174728Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 1.0000,  5.5000, 10.0000])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.linspace(1, 10, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:01:34.801399Z",
     "start_time": "2020-12-09T02:01:34.793420Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.5418, -0.9198, -1.0864],\n",
       "        [-1.8070,  1.8021,  0.3956]])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.randn(2, 3, device=t.device('cpu'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:01:40.698714Z",
     "start_time": "2020-12-09T02:01:40.693729Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0, 3, 4, 2, 1])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.randperm(5) # 长度为5的随机排列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:01:53.155512Z",
     "start_time": "2020-12-09T02:01:53.146507Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 0, 0],\n",
       "        [0, 1, 0]], dtype=torch.int32)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.eye(2, 3, dtype=t.int) # 对角线为1, 不要求行列数一致"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`torch.tensor`是在0.4版本新增加的一个新版本的创建tensor方法，使用的方法，和参数几乎和`np.array`完全一致"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:02:14.239469Z",
     "start_time": "2020-12-09T02:02:14.231462Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scalar: tensor(3.1416), shape of sclar: torch.Size([])\n"
     ]
    }
   ],
   "source": [
    "scalar = t.tensor(3.14159) \n",
    "print('scalar: %s, shape of sclar: %s' %(scalar, scalar.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:02:28.827138Z",
     "start_time": "2020-12-09T02:02:28.822151Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vector: tensor([1, 2]), shape of vector: torch.Size([2])\n"
     ]
    }
   ],
   "source": [
    "vector = t.tensor([1, 2])\n",
    "print('vector: %s, shape of vector: %s' %(vector, vector.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:02:42.682354Z",
     "start_time": "2020-12-09T02:02:42.678322Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 2])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor = t.Tensor(1,2) # 注意和t.tensor([1, 2])的区别\n",
    "tensor.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:02:53.517516Z",
     "start_time": "2020-12-09T02:02:53.511562Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[0.1000, 1.2000],\n",
       "         [2.2000, 3.1000],\n",
       "         [4.9000, 5.2000]]),\n",
       " torch.Size([3, 2]))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "matrix = t.tensor([[0.1, 1.2], [2.2, 3.1], [4.9, 5.2]])\n",
    "matrix,matrix.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:03:08.929910Z",
     "start_time": "2020-12-09T02:03:08.918937Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.1111, 0.2222, 0.3333]], dtype=torch.float64)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.tensor([[0.11111, 0.222222, 0.3333333]],\n",
    "                     dtype=t.float64,\n",
    "                     device=t.device('cpu'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:03:16.218812Z",
     "start_time": "2020-12-09T02:03:16.213824Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([0])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "empty_tensor = t.tensor([])\n",
    "empty_tensor.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 常用Tensor操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过`tensor.view`方法可以调整tensor的形状，但必须保证调整前后元素总数一致。`view`不会修改自身的数据，返回的新tensor与源tensor共享内存，也即更改其中的一个，另外一个也会跟着改变。在实际应用中可能经常需要添加或减少某一维度，这时候`squeeze`和`unsqueeze`两个函数就派上用场了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:04:29.036225Z",
     "start_time": "2020-12-09T02:04:29.031237Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 1, 2],\n",
       "        [3, 4, 5]])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6)\n",
    "a.view(2, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:04:36.234448Z",
     "start_time": "2020-12-09T02:04:36.230429Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 3])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.view(-1, 3) # 当某一维为-1的时候，会自动计算它的大小\n",
    "b.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:05:02.399169Z",
     "start_time": "2020-12-09T02:05:02.394154Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 1, 3])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.unsqueeze(1) # 注意形状，在第1维（下标从0开始）上增加“１” \n",
    "#等价于 b[:,None]\n",
    "b[:, None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:05:19.355116Z",
     "start_time": "2020-12-09T02:05:19.350095Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[0, 1, 2]],\n",
       "\n",
       "        [[3, 4, 5]]])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.unsqueeze(-2) # -2表示倒数第二个维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:05:47.694783Z",
     "start_time": "2020-12-09T02:05:47.686777Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[[0, 1, 2],\n",
       "          [3, 4, 5]]]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = b.view(1, 1, 1, 2, 3)\n",
    "c.squeeze(0) # 压缩第0维的“１”"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:06:04.907332Z",
     "start_time": "2020-12-09T02:06:04.902346Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 1, 1, 2, 3])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:06:18.797966Z",
     "start_time": "2020-12-09T02:06:18.789955Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 1, 2],\n",
       "        [3, 4, 5]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.squeeze() # 把所有维度为“1”的压缩"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:06:31.202554Z",
     "start_time": "2020-12-09T02:06:31.194605Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0, 100,   2],\n",
       "        [  3,   4,   5]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[1] = 100\n",
    "b # a修改，b作为view之后的，也会跟着修改"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`resize`是另一种可用来调整`size`的方法，但与`view`不同，它可以修改tensor的大小。如果新大小超过了原大小，会自动分配新的内存空间，而如果新大小小于原大小，则之前的数据依旧会被保存，看一个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:07:03.434321Z",
     "start_time": "2020-12-09T02:07:03.430302Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0, 100,   2]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.resize_(1, 3)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:07:19.434308Z",
     "start_time": "2020-12-09T02:07:19.431286Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[              0,             100,               2],\n",
       "        [              3,               4,               5],\n",
       "        [111295487561710,    420906860300,  86011015090012]])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.resize_(3, 3) # 旧的数据依旧保存着，多出的大小会分配新空间\n",
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 索引操作\n",
    "\n",
    "Tensor支持与numpy.ndarray类似的索引操作，语法上也类似，下面通过一些例子，讲解常用的索引操作。如无特殊说明，索引出来的结果与原tensor共享内存，也即修改一个，另一个会跟着修改。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:07:52.011069Z",
     "start_time": "2020-12-09T02:07:52.006054Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.1427, -0.6247,  0.7083, -0.8276],\n",
       "        [ 0.5155,  0.3117,  1.0403,  1.2435],\n",
       "        [ 1.3579, -0.9750, -1.5401,  0.7008]])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.randn(3, 4)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:08:03.642511Z",
     "start_time": "2020-12-09T02:08:03.638540Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-1.1427, -0.6247,  0.7083, -0.8276])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0] # 第0行(下标从0开始)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:08:08.494189Z",
     "start_time": "2020-12-09T02:08:08.490199Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-1.1427,  0.5155,  1.3579])"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:, 0] # 第0列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:08:15.594678Z",
     "start_time": "2020-12-09T02:08:15.590660Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0.7083)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0][2] # 第0行第2个元素，等价于a[0, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:08:21.627478Z",
     "start_time": "2020-12-09T02:08:21.622490Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(-0.8276)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0, -1] # 第0行最后一个元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:08:27.227271Z",
     "start_time": "2020-12-09T02:08:27.222255Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.1427, -0.6247,  0.7083, -0.8276],\n",
       "        [ 0.5155,  0.3117,  1.0403,  1.2435]])"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:2] # 前两行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:08:34.578442Z",
     "start_time": "2020-12-09T02:08:34.570434Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.1427, -0.6247],\n",
       "        [ 0.5155,  0.3117]])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:2, 0:2] # 前两行，第0,1列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:08:41.627347Z",
     "start_time": "2020-12-09T02:08:41.622361Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[-1.1427, -0.6247]])\n",
      "tensor([-1.1427, -0.6247])\n"
     ]
    }
   ],
   "source": [
    "print(a[0:1, :2]) # 第0行，前两列 \n",
    "print(a[0, :2]) # 注意两者的区别：形状不同"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:11:09.470884Z",
     "start_time": "2020-12-09T02:11:09.466895Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 3, 4])"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# None类似于np.newaxis, 为a新增了一个轴\n",
    "# 等价于a.view(1, a.shape[0], a.shape[1])\n",
    "a[None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:11:47.327401Z",
     "start_time": "2020-12-09T02:11:47.324377Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([1, 3, 4])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[None].shape # 等价于a[None,:,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:11:54.285006Z",
     "start_time": "2020-12-09T02:11:54.279053Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 1, 4])"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:,None,:].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:12:16.827928Z",
     "start_time": "2020-12-09T02:12:16.822913Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 1, 4, 1, 1])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[:,None,:,None,None].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:12:23.090601Z",
     "start_time": "2020-12-09T02:12:23.083591Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[False, False, False, False],\n",
       "        [False, False,  True,  True],\n",
       "        [ True, False, False, False]])"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a > 1 # 返回一个ByteTensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:12:36.160774Z",
     "start_time": "2020-12-09T02:12:36.150801Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.0403, 1.2435, 1.3579])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[a>1] # 等价于a.masked_select(a>1)\n",
    "# 选择结果与原tensor不共享内存空间"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:12:45.227822Z",
     "start_time": "2020-12-09T02:12:45.222807Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.1427, -0.6247,  0.7083, -0.8276],\n",
       "        [ 0.5155,  0.3117,  1.0403,  1.2435]])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[t.LongTensor([0,1])] # 第0行和第1行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其它常用的选择函数如表3-2所示。\n",
    "\n",
    "表3-2常用的选择函数\n",
    "\n",
    "函数|功能|\n",
    ":---:|:---:|\n",
    "index_select(input, dim, index)|在指定维度dim上选取，比如选取某些行、某些列\n",
    "masked_select(input, mask)|例子如上，a[a>0]，使用ByteTensor进行选取\n",
    "non_zero(input)|非0元素的下标\n",
    "gather(input, dim, index)|根据index，在dim维度上选取数据，输出的size与index一样\n",
    "\n",
    "\n",
    "`gather`是一个比较复杂的操作，对一个2维tensor，输出的每个元素如下：\n",
    "\n",
    "```python\n",
    "out[i][j] = input[index[i][j]][j]  # dim=0\n",
    "out[i][j] = input[i][index[i][j]]  # dim=1\n",
    "```\n",
    "三维tensor的`gather`操作同理，下面举几个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:14:20.299926Z",
     "start_time": "2020-12-09T02:14:20.294939Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  1,  2,  3],\n",
       "        [ 4,  5,  6,  7],\n",
       "        [ 8,  9, 10, 11],\n",
       "        [12, 13, 14, 15]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 16).view(4, 4)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:14:29.123084Z",
     "start_time": "2020-12-09T02:14:29.114107Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  5, 10, 15]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取对角线的元素\n",
    "index = t.LongTensor([[0,1,2,3]])\n",
    "a.gather(0, index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:14:55.531825Z",
     "start_time": "2020-12-09T02:14:55.527835Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 3],\n",
       "        [ 6],\n",
       "        [ 9],\n",
       "        [12]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取反对角线上的元素\n",
    "index = t.LongTensor([[3,2,1,0]]).t()\n",
    "a.gather(1, index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:14:58.447604Z",
     "start_time": "2020-12-09T02:14:58.442617Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  1,  2,  3],\n",
       "        [ 4,  5,  6,  7],\n",
       "        [ 8,  9, 10, 11],\n",
       "        [12, 13, 14, 15]])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:15:22.463625Z",
     "start_time": "2020-12-09T02:15:22.458638Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[12,  9,  6,  3]])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取反对角线上的元素，注意与上面的不同\n",
    "index = t.LongTensor([[3,2,1,0]])\n",
    "a.gather(0, index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:15:47.611835Z",
     "start_time": "2020-12-09T02:15:47.606860Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  3],\n",
       "        [ 5,  6],\n",
       "        [10,  9],\n",
       "        [15, 12]])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取两个对角线上的元素\n",
    "index = t.LongTensor([[0,1,2,3],[3,2,1,0]]).t()\n",
    "b = a.gather(1, index)\n",
    "b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "与`gather`相对应的逆操作是`scatter_`，`gather`把数据从input中按index取出，而`scatter_`是把取出的数据再放回去。注意`scatter_`函数是inplace操作。\n",
    "\n",
    "```python\n",
    "out = input.gather(dim, index)\n",
    "-->近似逆操作\n",
    "out = Tensor()\n",
    "out.scatter_(dim, index)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:17:11.535802Z",
     "start_time": "2020-12-09T02:17:11.526825Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  0.,  0.,  3.],\n",
       "        [ 0.,  5.,  6.,  0.],\n",
       "        [ 0.,  9., 10.,  0.],\n",
       "        [12.,  0.,  0., 15.]])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 把两个对角线元素放回去到指定位置\n",
    "c = t.zeros(4,4)\n",
    "c.scatter_(1, index, b.float())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对tensor的任何索引操作仍是一个tensor，想要获取标准的python对象数值，需要调用`tensor.item()`, 这个方法只对包含一个元素的tensor适用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:19:46.861846Z",
     "start_time": "2020-12-09T02:19:46.856829Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(0)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0,0] #依旧是tensor）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:19:52.057771Z",
     "start_time": "2020-12-09T02:19:52.054749Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0,0].item() # python float"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:19:58.543885Z",
     "start_time": "2020-12-09T02:19:58.539896Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([1, 1, 1])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = a[0:1, 0:1, None]\n",
    "print(d.shape)\n",
    "d.item() # 只包含一个元素的tensor即可调用tensor.item,与形状无关"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:20:28.012944Z",
     "start_time": "2020-12-09T02:20:27.947120Z"
    }
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "only one element tensors can be converted to Python scalars",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-59-ed6714e23428>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0ma\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mitem\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[1;31m# raise ValueError: only one element tensors can be converted to Python scalars\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mValueError\u001b[0m: only one element tensors can be converted to Python scalars"
     ]
    }
   ],
   "source": [
    "a[0].item()\n",
    "# raise ValueError: only one element tensors can be converted to Python scalars"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 高级索引\n",
    "PyTorch在0.2版本中完善了索引操作，目前已经支持绝大多数numpy的高级索引[^10]。高级索引可以看成是普通索引操作的扩展，但是高级索引操作的结果一般不和原始的Tensor共享内存。 \n",
    "[^10]: https://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:21:11.483892Z",
     "start_time": "2020-12-09T02:21:11.478877Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0,  1,  2],\n",
       "         [ 3,  4,  5],\n",
       "         [ 6,  7,  8]],\n",
       "\n",
       "        [[ 9, 10, 11],\n",
       "         [12, 13, 14],\n",
       "         [15, 16, 17]],\n",
       "\n",
       "        [[18, 19, 20],\n",
       "         [21, 22, 23],\n",
       "         [24, 25, 26]]])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.arange(0,27).view(3,3,3)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:21:22.673439Z",
     "start_time": "2020-12-09T02:21:22.666457Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([14, 24])"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[[1, 2], [1, 2], [2, 0]] # x[1,1,2]和x[2,2,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:21:28.973746Z",
     "start_time": "2020-12-09T02:21:28.966734Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([19, 10,  1])"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[[2, 1, 0], [0], [1]] # x[2,0,1],x[1,0,1],x[0,0,1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:21:39.902983Z",
     "start_time": "2020-12-09T02:21:39.897967Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 0,  1,  2],\n",
       "         [ 3,  4,  5],\n",
       "         [ 6,  7,  8]],\n",
       "\n",
       "        [[18, 19, 20],\n",
       "         [21, 22, 23],\n",
       "         [24, 25, 26]]])"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x[[0, 2], ...] # x[0] 和 x[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tensor类型\n",
    "\n",
    "Tensor有不同的数据类型，如表3-3所示，每种类型分别对应有CPU和GPU版本(HalfTensor除外)。默认的tensor是FloatTensor，可通过`t.set_default_tensor_type` 来修改默认tensor类型(如果默认类型为GPU tensor，则所有操作都将在GPU上进行)。Tensor的类型对分析内存占用很有帮助。例如对于一个size为(1000, 1000, 1000)的FloatTensor，它有`1000*1000*1000=10^9`个元素，每个元素占32bit/8 = 4Byte内存，所以共占大约4GB内存/显存。HalfTensor是专门为GPU版本设计的，同样的元素个数，显存占用只有FloatTensor的一半，所以可以极大缓解GPU显存不足的问题，但由于HalfTensor所能表示的数值大小和精度有限[^2]，所以可能出现溢出等问题。\n",
    "\n",
    "[^2]: https://stackoverflow.com/questions/872544/what-range-of-numbers-can-be-represented-in-a-16-32-and-64-bit-ieee-754-syste\n",
    "\n",
    "表3-3: tensor数据类型\n",
    "\n",
    "| Data type                | dtype                             | CPU tensor                                                   | GPU tensor                |\n",
    "| ------------------------ | --------------------------------- | ------------------------------------------------------------ | ------------------------- |\n",
    "| 32-bit floating point    | `torch.float32` or `torch.float`  | `torch.FloatTensor`                                          | `torch.cuda.FloatTensor`  |\n",
    "| 64-bit floating point    | `torch.float64` or `torch.double` | `torch.DoubleTensor`                                         | `torch.cuda.DoubleTensor` |\n",
    "| 16-bit floating point    | `torch.float16` or `torch.half`   | `torch.HalfTensor`                                           | `torch.cuda.HalfTensor`   |\n",
    "| 8-bit integer (unsigned) | `torch.uint8`                     | [`torch.ByteTensor`](https://pytorch.org/docs/stable/tensors.html#torch.ByteTensor) | `torch.cuda.ByteTensor`   |\n",
    "| 8-bit integer (signed)   | `torch.int8`                      | `torch.CharTensor`                                           | `torch.cuda.CharTensor`   |\n",
    "| 16-bit integer (signed)  | `torch.int16` or `torch.short`    | `torch.ShortTensor`                                          | `torch.cuda.ShortTensor`  |\n",
    "| 32-bit integer (signed)  | `torch.int32` or `torch.int`      | `torch.IntTensor`                                            | `torch.cuda.IntTensor`    |\n",
    "| 64-bit integer (signed)  | `torch.int64` or `torch.long`     | `torch.LongTensor`                                           | `torch.cuda.LongTensor`   |\n",
    "\n",
    " \n",
    "\n",
    "各数据类型之间可以互相转换，`type(new_type)`是通用的做法，同时还有`float`、`long`、`half`等快捷方法。CPU tensor与GPU tensor之间的互相转换通过`tensor.cuda`和`tensor.cpu`方法实现，此外还可以使用`tensor.to(device)`。Tensor还有一个`new`方法，用法与`t.Tensor`一样，会调用该tensor对应类型的构造函数，生成与当前tensor类型一致的tensor。`torch.*_like(tensora)` 可以生成和`tensora`拥有同样属性(类型，形状，cpu/gpu)的新tensor。 `tensor.new_*(new_shape)` 新建一个不同形状的tensor。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:24:11.690190Z",
     "start_time": "2020-12-09T02:24:11.687198Z"
    }
   },
   "outputs": [],
   "source": [
    "# 设置默认tensor，注意参数是字符串\n",
    "t.set_default_tensor_type('torch.DoubleTensor')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:24:26.058393Z",
     "start_time": "2020-12-09T02:24:26.055373Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float64"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.Tensor(2,3)\n",
    "a.dtype # 现在a是DoubleTensor,dtype是float64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:24:35.627218Z",
     "start_time": "2020-12-09T02:24:35.623200Z"
    }
   },
   "outputs": [],
   "source": [
    "# 恢复之前的默认设置\n",
    "t.set_default_tensor_type('torch.FloatTensor')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:24:44.937875Z",
     "start_time": "2020-12-09T02:24:44.934883Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 把a转成FloatTensor，等价于b=a.type(t.FloatTensor)\n",
    "b = a.float() \n",
    "b.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:24:54.815000Z",
     "start_time": "2020-12-09T02:24:54.807021Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.],\n",
       "        [0., 0., 0.]])"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = a.type_as(b)\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:25:13.356411Z",
     "start_time": "2020-12-09T02:25:13.351455Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.],\n",
       "        [0., 0., 0.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.new(2,3) # 等价于torch.DoubleTensor(2,3)，建议使用a.new_tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:25:31.037795Z",
     "start_time": "2020-12-09T02:25:31.030785Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0.],\n",
       "        [0., 0., 0.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.zeros_like(a) #等价于t.zeros(a.shape,dtype=a.dtype,device=a.device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:25:43.054332Z",
     "start_time": "2020-12-09T02:25:43.049346Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0, 0, 0],\n",
       "        [0, 0, 0]], dtype=torch.int16)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.zeros_like(a, dtype=t.int16) #可以修改某些属性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:25:48.992154Z",
     "start_time": "2020-12-09T02:25:48.984205Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.2752, 0.2905, 0.4608],\n",
       "        [0.5302, 0.7321, 0.0557]], dtype=torch.float64)"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.rand_like(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:25:56.300510Z",
     "start_time": "2020-12-09T02:25:56.295494Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1, 1, 1, 1, 1],\n",
       "        [1, 1, 1, 1, 1],\n",
       "        [1, 1, 1, 1, 1],\n",
       "        [1, 1, 1, 1, 1]], dtype=torch.int32)"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.new_ones(4,5, dtype=t.int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:26:03.358710Z",
     "start_time": "2020-12-09T02:26:03.350732Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3., 4.], dtype=torch.float64)"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.new_tensor([3,4]) # "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 逐元素操作\n",
    "\n",
    "这部分操作会对tensor的每一个元素(point-wise，又名element-wise)进行操作，此类操作的输入与输出形状一致。常用的操作如表3-4所示。\n",
    "\n",
    "表3-4: 常见的逐元素操作\n",
    "\n",
    "|函数|功能|\n",
    "|:--:|:--:|\n",
    "|abs/sqrt/div/exp/fmod/log/pow..|绝对值/平方根/除法/指数/求余/求幂..|\n",
    "|cos/sin/asin/atan2/cosh..|相关三角函数|\n",
    "|ceil/round/floor/trunc| 上取整/四舍五入/下取整/只保留整数部分|\n",
    "|clamp(input, min, max)|超过min和max部分截断|\n",
    "|sigmod/tanh..|激活函数\n",
    "\n",
    "对于很多操作，例如div、mul、pow、fmod等，PyTorch都实现了运算符重载，所以可以直接使用运算符。如`a ** 2` 等价于`torch.pow(a,2)`, `a * 2`等价于`torch.mul(a,2)`。\n",
    "\n",
    "其中`clamp(x, min, max)`的输出满足以下公式：\n",
    "$$\n",
    "y_i =\n",
    "\\begin{cases}\n",
    "min,  & \\text{if  } x_i \\lt min \\\\\n",
    "x_i,  & \\text{if  } min \\le x_i \\le max  \\\\\n",
    "max,  & \\text{if  } x_i \\gt max\\\\\n",
    "\\end{cases}\n",
    "$$\n",
    "`clamp`常用在某些需要比较大小的地方，如取一个tensor的每个元素与另一个数的较大值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:27:36.821663Z",
     "start_time": "2020-12-09T02:27:36.807700Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.0000,  0.5403, -0.4161],\n",
       "        [-0.9900, -0.6536,  0.2837]])"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6).view(2, 3).float()\n",
    "t.cos(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:27:45.360261Z",
     "start_time": "2020-12-09T02:27:45.350318Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 1., 2.],\n",
       "        [0., 1., 2.]])"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a % 3 # 等价于t.fmod(a, 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:27:50.801609Z",
     "start_time": "2020-12-09T02:27:50.791665Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  1.,  4.],\n",
       "        [ 9., 16., 25.]])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a ** 2 # 等价于t.pow(a, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:27:57.008547Z",
     "start_time": "2020-12-09T02:27:56.998573Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0., 1., 2.],\n",
      "        [3., 4., 5.]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[3., 3., 3.],\n",
       "        [3., 4., 5.]])"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 取a中的每一个元素与3相比较大的一个 (小于3的截断成3)\n",
    "print(a)\n",
    "t.clamp(a, min=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:28:09.155399Z",
     "start_time": "2020-12-09T02:28:09.146394Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000,  0.8415,  0.9093],\n",
       "        [ 0.1411, -0.7568, -0.9589]])"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.sin_() # 效果同 a = a.sin();b=a ,但是更高效节省显存\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  归并操作 \n",
    "此类操作会使输出形状小于输入形状，并可以沿着某一维度进行指定操作。如加法`sum`，既可以计算整个tensor的和，也可以计算tensor中每一行或每一列的和。常用的归并操作如表3-5所示。\n",
    "\n",
    "表3-5: 常用归并操作\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|mean/sum/median/mode|均值/和/中位数/众数|\n",
    "|norm/dist|范数/距离|\n",
    "|std/var|标准差/方差|\n",
    "|cumsum/cumprod|累加/累乘|\n",
    "\n",
    "以上大多数函数都有一个参数**`dim`**，用来指定这些操作是在哪个维度上执行的。关于dim(对应于Numpy中的axis)的解释众说纷纭，这里提供一个简单的记忆方式：\n",
    "\n",
    "假设输入的形状是(m, n, k)\n",
    "\n",
    "- 如果指定dim=0，输出的形状就是(1, n, k)或者(n, k)\n",
    "- 如果指定dim=1，输出的形状就是(m, 1, k)或者(m, k)\n",
    "- 如果指定dim=2，输出的形状就是(m, n, 1)或者(m, n)\n",
    "\n",
    "size中是否有\"1\"，取决于参数`keepdim`，`keepdim=True`会保留维度`1`。注意，以上只是经验总结，并非所有函数都符合这种形状变化方式，如`cumsum`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:32:23.613775Z",
     "start_time": "2020-12-09T02:32:23.607760Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[2., 2., 2.]])"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.ones(2, 3)\n",
    "b.sum(dim = 0, keepdim=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:32:32.731801Z",
     "start_time": "2020-12-09T02:32:32.726782Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2., 2., 2.])"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# keepdim=False，不保留维度\"1\"，注意形状\n",
    "b.sum(dim=0, keepdim=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:32:40.140204Z",
     "start_time": "2020-12-09T02:32:40.134255Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([3., 3.])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.sum(dim=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:32:47.311426Z",
     "start_time": "2020-12-09T02:32:47.302421Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0, 1, 2],\n",
      "        [3, 4, 5]])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[ 0,  1,  3],\n",
       "        [ 3,  7, 12]])"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6).view(2, 3)\n",
    "print(a)\n",
    "a.cumsum(dim=1) # 沿着行累加"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 比较\n",
    "比较函数中有一些是逐元素比较，操作类似于逐元素操作，还有一些则类似于归并操作。常用比较函数如表3-6所示。\n",
    "\n",
    "表3-6: 常用比较函数\n",
    "\n",
    "|函数|功能|\n",
    "|:--:|:--:|\n",
    "|gt/lt/ge/le/eq/ne|大于/小于/大于等于/小于等于/等于/不等|\n",
    "|topk|最大的k个数|\n",
    "|sort|排序|\n",
    "|max/min|比较两个tensor最大最小值|\n",
    "\n",
    "表中第一行的比较操作已经实现了运算符重载，因此可以使用`a>=b`、`a>b`、`a!=b`、`a==b`，其返回结果是一个`ByteTensor`，可用来选取元素。max/min这两个操作比较特殊，以max来说，它有以下三种使用情况：\n",
    "- t.max(tensor)：返回tensor中最大的一个数\n",
    "- t.max(tensor,dim)：指定维上最大的数，返回tensor和下标\n",
    "- t.max(tensor1, tensor2): 比较两个tensor相比较大的元素\n",
    "\n",
    "至于比较一个tensor和一个数，可以使用clamp函数。下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:34:26.045370Z",
     "start_time": "2020-12-09T02:34:26.040413Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  3.,  6.],\n",
       "        [ 9., 12., 15.]])"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.linspace(0, 15, 6).view(2, 3)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:34:45.531395Z",
     "start_time": "2020-12-09T02:34:45.527399Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[15., 12.,  9.],\n",
       "        [ 6.,  3.,  0.]])"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.linspace(15, 0, 6).view(2, 3)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:34:59.933384Z",
     "start_time": "2020-12-09T02:34:59.927408Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[False, False, False],\n",
       "        [ True,  True,  True]])"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a>b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:35:08.446523Z",
     "start_time": "2020-12-09T02:35:08.442505Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([ 9., 12., 15.])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[a>b] # a中大于b的元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:35:14.123860Z",
     "start_time": "2020-12-09T02:35:14.119871Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(15.)"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.max(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:35:23.137888Z",
     "start_time": "2020-12-09T02:35:23.129909Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.return_types.max(\n",
       "values=tensor([15.,  6.]),\n",
       "indices=tensor([0, 0]))"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.max(b, dim=1) \n",
    "# 第一个返回值的15和6分别表示第0行和第1行最大的元素\n",
    "# 第二个返回值的0和0表示上述最大的数是该行第0个元素"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:35:35.375539Z",
     "start_time": "2020-12-09T02:35:35.367561Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[15., 12.,  9.],\n",
       "        [ 9., 12., 15.]])"
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.max(a,b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:35:43.837921Z",
     "start_time": "2020-12-09T02:35:43.832934Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[10., 10., 10.],\n",
       "        [10., 12., 15.]])"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 比较a和10较大的元素\n",
    "t.clamp(a, min=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 线性代数\n",
    "\n",
    "PyTorch的线性函数主要封装了Blas和Lapack，其用法和接口都与之类似。常用的线性代数函数如表3-7所示。\n",
    "\n",
    "表3-7: 常用的线性代数函数\n",
    "\n",
    "|函数|功能|\n",
    "|:---:|:---:|\n",
    "|trace|对角线元素之和(矩阵的迹)|\n",
    "|diag|对角线元素|\n",
    "|triu/tril|矩阵的上三角/下三角，可指定偏移量|\n",
    "|mm/bmm|矩阵乘法，batch的矩阵乘法|\n",
    "|addmm/addbmm/addmv/addr/badbmm..|矩阵运算\n",
    "|t|转置|\n",
    "|dot/cross|内积/外积\n",
    "|inverse|求逆矩阵\n",
    "|svd|奇异值分解\n",
    "\n",
    "具体使用说明请参见官方文档[^3]，需要注意的是，矩阵的转置会导致存储空间不连续，需调用它的`.contiguous`方法将其转为连续。\n",
    "[^3]: http://pytorch.org/docs/torch.html#blas-and-lapack-operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:36:40.156999Z",
     "start_time": "2020-12-09T02:36:40.152016Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.t()\n",
    "b.is_contiguous()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:36:47.387275Z",
     "start_time": "2020-12-09T02:36:47.383283Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.,  9.],\n",
       "        [ 3., 12.],\n",
       "        [ 6., 15.]])"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.contiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.2 Tensor和Numpy\n",
    "\n",
    "Tensor和Numpy数组之间具有很高的相似性，彼此之间的互操作也非常简单高效。需要注意的是，Numpy和Tensor共享内存。由于Numpy历史悠久，支持丰富的操作，所以当遇到Tensor不支持的操作时，可先转成Numpy数组，处理后再转回tensor，其转换开销很小。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:37:52.817884Z",
     "start_time": "2020-12-09T02:37:52.807882Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1., 1., 1.],\n",
       "       [1., 1., 1.]], dtype=float32)"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "a = np.ones([2, 3],dtype=np.float32)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:37:59.646541Z",
     "start_time": "2020-12-09T02:37:59.638597Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.from_numpy(a)\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:38:06.477913Z",
     "start_time": "2020-12-09T02:38:06.473895Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.Tensor(a) # 也可以直接将numpy对象传入Tensor\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:38:16.156496Z",
     "start_time": "2020-12-09T02:38:16.150512Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  1., 100.,   1.],\n",
       "        [  1.,   1.,   1.]])"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0, 1]=100\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:38:25.660122Z",
     "start_time": "2020-12-09T02:38:25.655136Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1., 100.,   1.],\n",
       "       [  1.,   1.,   1.]], dtype=float32)"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = b.numpy() # a, b, c三个对象共享内存\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意**： 当numpy的数据类型和Tensor的类型不一样的时候，数据会被复制，不会共享内存。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:38:45.709229Z",
     "start_time": "2020-12-09T02:38:45.704271Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dtype('float64')"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = np.ones([2, 3])\n",
    "# 注意和上面的a的区别（dtype不是float32）\n",
    "a.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:38:54.091238Z",
     "start_time": "2020-12-09T02:38:54.087277Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.float32"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.Tensor(a) # 此处进行拷贝，不共享内存\n",
    "b.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:39:07.708043Z",
     "start_time": "2020-12-09T02:39:07.704025Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = t.from_numpy(a) # 注意c的类型（DoubleTensor）\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:39:16.925504Z",
     "start_time": "2020-12-09T02:39:16.920518Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a[0, 1] = 100\n",
    "b # b与a不共享内存，所以即使a改变了，b也不变"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:39:25.134911Z",
     "start_time": "2020-12-09T02:39:25.129954Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  1., 100.,   1.],\n",
       "        [  1.,   1.,   1.]], dtype=torch.float64)"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c # c与a共享内存"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:39:36.364681Z",
     "start_time": "2020-12-09T02:39:36.359695Z"
    }
   },
   "source": [
    "**注意：** 不论输入的类型是什么，t.tensor都会进行数据拷贝，不会共享内存"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:39:45.627386Z",
     "start_time": "2020-12-09T02:39:45.624367Z"
    }
   },
   "outputs": [],
   "source": [
    "tensor = t.tensor(a) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:39:50.466206Z",
     "start_time": "2020-12-09T02:39:50.458229Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  1., 100.,   1.],\n",
       "       [  1.,   1.,   1.]])"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tensor[0,0]=0\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "广播法则(broadcast)是科学运算中经常使用的一个技巧，它在快速执行向量化的同时不会占用额外的内存/显存。\n",
    "Numpy的广播法则定义如下：\n",
    "\n",
    "- 让所有输入数组都向其中shape最长的数组看齐，shape中不足的部分通过在前面加1补齐\n",
    "- 两个数组要么在某一个维度的长度一致，要么其中一个为1，否则不能计算 \n",
    "- 当输入数组的某个维度的长度为1时，计算时沿此维度复制扩充成一样的形状\n",
    "\n",
    "PyTorch当前已经支持了自动广播法则，但是笔者还是建议读者通过以下两个函数的组合手动实现广播法则，这样更直观，更不易出错：\n",
    "\n",
    "- `unsqueeze`或者`view`，或者tensor[None],：为数据某一维的形状补1，实现法则1\n",
    "- `expand`或者`expand_as`，重复数组，实现法则3；该操作不会复制数组，所以不会占用额外的空间。\n",
    "\n",
    "注意，repeat实现与expand相类似的功能，但是repeat会把相同数据复制多份，因此会占用额外的空间。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:42:10.655448Z",
     "start_time": "2020-12-09T02:42:10.651490Z"
    }
   },
   "outputs": [],
   "source": [
    "a = t.ones(3, 2)\n",
    "b = t.zeros(2, 3,1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:42:31.121348Z",
     "start_time": "2020-12-09T02:42:31.116393Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1., 1.],\n",
       "         [1., 1.],\n",
       "         [1., 1.]],\n",
       "\n",
       "        [[1., 1.],\n",
       "         [1., 1.],\n",
       "         [1., 1.]]])"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 自动广播法则\n",
    "# 第一步：a是2维,b是3维，所以先在较小的a前面补1 ，\n",
    "#               即：a.unsqueeze(0)，a的形状变成（1，3，2），b的形状是（2，3，1）,\n",
    "# 第二步:   a和b在第一维和第三维形状不一样，其中一个为1 ，\n",
    "#               可以利用广播法则扩展，两个形状都变成了（2，3，2）\n",
    "a+b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:42:50.702252Z",
     "start_time": "2020-12-09T02:42:50.697232Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[1., 1.],\n",
       "         [1., 1.],\n",
       "         [1., 1.]],\n",
       "\n",
       "        [[1., 1.],\n",
       "         [1., 1.],\n",
       "         [1., 1.]]])"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 手动广播法则\n",
    "# 或者 a.view(1,3,2).expand(2,3,2)+b.expand(2,3,2)\n",
    "a[None].expand(2, 3, 2) + b.expand(2,3,2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:43:03.758519Z",
     "start_time": "2020-12-09T02:43:03.754559Z"
    }
   },
   "outputs": [],
   "source": [
    "# expand不会占用额外空间，只会在需要的时候才扩充，可极大节省内存\n",
    "e = a.unsqueeze(0).expand(10000000000000, 3,2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.3 内部结构\n",
    "\n",
    "tensor的数据结构如图3-1所示。tensor分为头信息区(Tensor)和存储区(Storage)，信息区主要保存着tensor的形状（size）、步长（stride）、数据类型（type）等信息，而真正的数据则保存成连续数组。由于数据动辄成千上万，因此信息区元素占用内存较少，主要内存占用则取决于tensor中元素的数目，也即存储区的大小。\n",
    "\n",
    "一般来说一个tensor有着与之相对应的storage, storage是在data之上封装的接口，便于使用，而不同tensor的头信息一般不同，但却可能使用相同的数据。下面看两个例子。\n",
    "\n",
    "![图3-1: Tensor的数据结构](imgs/tensor_data_structure.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:48:57.923419Z",
     "start_time": "2020-12-09T02:48:57.916437Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0\n",
       " 1\n",
       " 2\n",
       " 3\n",
       " 4\n",
       " 5\n",
       "[torch.LongStorage of size 6]"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 6)\n",
    "a.storage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:49:05.562751Z",
     "start_time": "2020-12-09T02:49:05.558766Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0\n",
       " 1\n",
       " 2\n",
       " 3\n",
       " 4\n",
       " 5\n",
       "[torch.LongStorage of size 6]"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = a.view(2, 3)\n",
    "b.storage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:49:33.951060Z",
     "start_time": "2020-12-09T02:49:33.947071Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 一个对象的id值可以看作它在内存中的地址\n",
    "# storage的内存地址一样，即是同一个storage\n",
    "id(b.storage()) == id(a.storage())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:49:41.135216Z",
     "start_time": "2020-12-09T02:49:41.131228Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[  0, 100,   2],\n",
       "        [  3,   4,   5]])"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# a改变，b也随之改变，因为他们共享storage\n",
    "a[1] = 100\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:49:49.103018Z",
     "start_time": "2020-12-09T02:49:49.099029Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       " 0\n",
       " 100\n",
       " 2\n",
       " 3\n",
       " 4\n",
       " 5\n",
       "[torch.LongStorage of size 6]"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c = a[2:] \n",
    "c.storage()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:50:10.571461Z",
     "start_time": "2020-12-09T02:50:10.568468Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2186159641808, 2186159641792)"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.data_ptr(), a.data_ptr() # data_ptr返回tensor首元素的内存地址\n",
    "# 可以看出相差8，这是因为2*4=8--相差两个元素，每个元素占4个字节(float)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:50:25.564265Z",
     "start_time": "2020-12-09T02:50:25.560280Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([   0,  100, -100,    3,    4,    5])"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c[0] = -100 # c[0]的内存地址对应a[2]的内存地址\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:51:31.756105Z",
     "start_time": "2020-12-09T02:51:31.752115Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[6666,  100, -100],\n",
       "        [   3,    4,    5]])"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = t.LongTensor(c.storage())\n",
    "d[0] = 6666\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:51:45.531351Z",
     "start_time": "2020-12-09T02:51:45.527334Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 下面４个tensor共享storage\n",
    "id(a.storage()) == id(b.storage()) == id(c.storage()) == id(d.storage())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:51:54.171912Z",
     "start_time": "2020-12-09T02:51:54.167953Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 2, 0)"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.storage_offset(), c.storage_offset(), d.storage_offset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:52:04.955772Z",
     "start_time": "2020-12-09T02:52:04.951782Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e = b[::2, ::2] # 隔2行/列取一个元素\n",
    "id(e.storage()) == id(a.storage())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:52:12.748685Z",
     "start_time": "2020-12-09T02:52:12.743727Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((3, 1), (6, 2))"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.stride(), e.stride()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:52:22.591197Z",
     "start_time": "2020-12-09T02:52:22.587237Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.is_contiguous()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:52:31.326245Z",
     "start_time": "2020-12-09T02:52:31.321258Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.is_contiguous()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见绝大多数操作并不修改tensor的数据，而只是修改了tensor的头信息。这种做法更节省内存，同时提升了处理速度。在使用中需要注意。\n",
    "此外有些操作会导致tensor不连续，这时需调用`tensor.contiguous`方法将它们变成连续的数据，该方法会使数据复制一份，不再与原来的数据共享storage。\n",
    "另外读者可以思考一下，之前说过的高级索引一般不共享stroage，而普通索引共享storage，这是为什么？（提示：普通索引可以通过只修改tensor的offset，stride和size，而不修改storage来实现）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.4 其它有关Tensor的话题\n",
    "这部分的内容不好专门划分一小节，但是笔者认为仍值得读者注意，故而将其放在这一小节。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GPU/CPU\n",
    "tensor可以很随意的在gpu/cpu上传输。使用`tensor.cuda(device_id)`或者`tensor.cpu()`。另外一个更通用的方法是`tensor.to(device)`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:53:48.530177Z",
     "start_time": "2020-12-09T02:53:48.525162Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "device(type='cpu')"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.randn(3, 4)\n",
    "a.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:53:54.818662Z",
     "start_time": "2020-12-09T02:53:54.808717Z"
    }
   },
   "outputs": [],
   "source": [
    "if t.cuda.is_available():\n",
    "    a = t.randn(3,4, device=t.device('cuda:1'))\n",
    "    # 等价于\n",
    "    # a.t.randn(3,4).cuda(1)\n",
    "    # 但是前者更快\n",
    "    a.device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:54:01.167396Z",
     "start_time": "2020-12-09T02:54:01.162409Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.1825,  0.5507,  1.0903, -0.4835],\n",
       "        [ 0.9243, -0.5512, -0.0497, -1.5169],\n",
       "        [ 1.1583,  0.1252, -0.9395, -0.4139]])"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "device = t.device('cpu')\n",
    "a.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**注意**\n",
    "- 尽量使用`tensor.to(device)`, 将`device`设为一个可配置的参数，这样可以很轻松的使程序同时兼容GPU和CPU\n",
    "- 数据在GPU之中传输的速度要远快于内存(CPU)到显存(GPU), 所以尽量避免频繁的在内存和显存中传输数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 持久化\n",
    "Tensor的保存和加载十分的简单，使用t.save和t.load即可完成相应的功能。在save/load时可指定使用的`pickle`模块，在load时还可将GPU tensor映射到CPU或其它GPU上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:54:45.083909Z",
     "start_time": "2020-12-09T02:54:45.079920Z"
    }
   },
   "outputs": [],
   "source": [
    "if t.cuda.is_available():\n",
    "    a = a.cuda(1) # 把a转为GPU1上的tensor,\n",
    "    t.save(a,'a.pth')\n",
    "\n",
    "    # 加载为b, 存储于GPU1上(因为保存时tensor就在GPU1上)\n",
    "    b = t.load('a.pth')\n",
    "    # 加载为c, 存储于CPU\n",
    "    c = t.load('a.pth', map_location=lambda storage, loc: storage)\n",
    "    # 加载为d, 存储于GPU0上\n",
    "    d = t.load('a.pth', map_location={'cuda:1':'cuda:0'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####   向量化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "向量化计算是一种特殊的并行计算方式，相对于一般程序在同一时间只执行一个操作的方式，它可在同一时间执行多个操作，通常是对不同的数据执行同样的一个或一批指令，或者说把指令应用于一个数组/向量上。向量化可极大提高科学运算的效率，Python本身是一门高级语言，使用很方便，但这也意味着很多操作很低效，尤其是`for`循环。在科学计算程序中应当极力避免使用Python原生的`for循环`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:55:56.138679Z",
     "start_time": "2020-12-09T02:55:56.135687Z"
    }
   },
   "outputs": [],
   "source": [
    "def for_loop_add(x, y):\n",
    "    result = []\n",
    "    for i,j in zip(x, y):\n",
    "        result.append(i + j)\n",
    "    return t.Tensor(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:56:02.363053Z",
     "start_time": "2020-12-09T02:56:02.295204Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "875 µs ± 115 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n",
      "The slowest run took 9.40 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
      "7.29 µs ± 9.69 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "x = t.zeros(100)\n",
    "y = t.ones(100)\n",
    "%timeit -n 10 for_loop_add(x, y)\n",
    "%timeit -n 10 x + y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见二者有超过几十倍的速度差距，因此在实际使用中应尽量调用内建函数(buildin-function)，这些函数底层由C/C++实现，能通过执行底层优化实现高效计算。因此在平时写代码时，就应养成向量化的思维习惯，千万避免对较大的tensor进行逐元素遍历。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此外还有以下几点需要注意：\n",
    "- 大多数`t.function`都有一个参数`out`，这时候产生的结果将保存在out指定tensor之中。\n",
    "- `t.set_num_threads`可以设置PyTorch进行CPU多线程并行计算时候所占用的线程数，这个可以用来限制PyTorch所占用的CPU数目。\n",
    "- `t.set_printoptions`可以用来设置打印tensor时的数值精度和格式。\n",
    "下面举例说明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:57:22.123432Z",
     "start_time": "2020-12-09T02:57:22.056611Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor(19999999) tensor(19999998)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(tensor(19999999), tensor(19999998))"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.arange(0, 20000000)\n",
    "print(a[-1], a[-2]) # 32bit的IntTensor精度有限导致溢出\n",
    "b = t.LongTensor()\n",
    "t.arange(0, 20000000, out=b) # 64bit的LongTensor不会溢出\n",
    "b[-1],b[-2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:58:15.721508Z",
     "start_time": "2020-12-09T02:58:15.703526Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.7816, -0.8966,  1.0354],\n",
       "        [ 0.7441,  0.8365, -1.3186]])"
      ]
     },
     "execution_count": 134,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.randn(2,3)\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T02:58:23.811243Z",
     "start_time": "2020-12-09T02:58:23.806228Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.7816215158, -0.8966382146,  1.0354435444],\n",
       "        [ 0.7441257834,  0.8365046382, -1.3185501099]])"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t.set_printoptions(precision=10)\n",
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1.5 小试牛刀：线性回归"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "线性回归是机器学习入门知识，应用十分广泛。线性回归利用数理统计中回归分析，来确定两种或两种以上变量间相互依赖的定量关系的，其表达形式为$y = wx+b+e$，$e$为误差服从均值为0的正态分布。首先让我们来确认线性回归的损失函数：\n",
    "$$\n",
    "loss = \\sum_i^N \\frac 1 2 ({y_i-(wx_i+b)})^2\n",
    "$$\n",
    "然后利用随机梯度下降法更新参数$\\textbf{w}$和$\\textbf{b}$来最小化损失函数，最终学得$\\textbf{w}$和$\\textbf{b}$的数值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:01:31.294223Z",
     "start_time": "2020-12-09T03:01:30.377446Z"
    }
   },
   "outputs": [],
   "source": [
    "import torch as t\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from IPython import display\n",
    "\n",
    "device = t.device('cpu') #如果你想用gpu，改成t.device('cuda:0')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:01:44.685482Z",
     "start_time": "2020-12-09T03:01:44.680184Z"
    }
   },
   "outputs": [],
   "source": [
    "# 设置随机数种子，保证在不同电脑上运行时下面的输出一致\n",
    "t.manual_seed(1000) \n",
    "\n",
    "def get_fake_data(batch_size=8):\n",
    "    ''' 产生随机数据：y=x*2+3，加上了一些噪声'''\n",
    "    x = t.rand(batch_size, 1, device=device) * 5\n",
    "    y = x * 2 + 3 +  t.randn(batch_size, 1, device=device)\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:01:57.928324Z",
     "start_time": "2020-12-09T03:01:57.771744Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1fd02fa1c88>"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWwAAAD4CAYAAADIH9xYAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAP+ElEQVR4nO3dX2xk513G8efBcZRJ2sooO0DsJGwrVebPrhKvRquUiCiQgNM0apZVL4LUIirEAgqQVMKoywVVr/bCCPHnArRqi4ooQSXdWFXU1OkFoepFUmbjpLvbjSENpM24kAnFTUNGzcb8uPDYsSe258zOnJnzHn8/krXjM++Of2ff3WfOvud953VECABQfD8y6gIAANkQ2ACQCAIbABJBYANAIghsAEjEFXm86IEDB+LgwYN5vDQAlNLZs2dfiYjqXm1yCeyDBw+qXq/n8dIAUEq2X+zWhiERAEgEgQ0AiSCwASARBDYAJILABoBE5DJLBAD2k4WlhuYXl7Wy2tLkREVzs9M6NjM18J9DYANAHxaWGjp55pxal9YkSY3Vlk6eOSdJAw9thkQAoA/zi8ubYb2hdWlN84vLA/9ZBDYA9GFltdXT8X4Q2ADQh8mJSk/H+0FgA0Af5manVRkf23asMj6mudnpgf8sbjoCQB82biwySwQAEnBsZiqXgO7EkAgAJILABoBEENgAkAjGsAEU2rCWfaeAwAZQWMNc9p0ChkQAFNYwl32nIFNg237A9nnbF2w/mHNNACBpuMu+U9A1sG0fkvSbko5KuknSPbbfm3dhADDMZd8pyHKF/dOSnoyI1yPiTUn/LOlX8i0LAIa77DsFWQL7vKTbbF9r+2pJd0u6obOR7RO267brzWZz0HUC2IeOzUzp1PHDmpqoyJKmJio6dfzwvrzhKEmOiO6N7N+QdL+k1yR9U1IrIj62W/tarRb1en1gRQLAbsoy7c/22Yio7dUm003HiPh0RByJiNskfU/Svw2iQADox8a0v8ZqS6G3pv0tLDVGXVouss4S+bH2rzdKOi7poTyLAoAs9tu0v6wLZ75g+1pJlyTdHxH/k2NNAJDJfpv2lymwI+Ln8y4EAHo1OVFRY4dwLuu0P1Y6AkjWfpv2x2eJAEjWMHd7KQICG0DShrXbSxEwJAIAieAKG0AplWVBzVYENoDSKevnaDMkAqB0yrqghsAGUDplXVBDYAMonbJ+jjaBDaB0yrqghpuOAEqnrAtqCGwApVTGBTUMiQBAIghsAEgEgQ0AiSCwASARBDYAJILABoBEENgAkAgCGwASkSmwbX/M9gXb520/ZPuqvAsDAGzXNbBtT0n6fUm1iDgkaUzSfXkXBgDYLuuQyBWSKravkHS1pJX8SgIA7KTrZ4lERMP2n0j6tqSWpMcj4vHOdrZPSDohSTfeeOOg6wSwgzJug4XdZRkS+VFJ90p6t6RJSdfY/nBnu4g4HRG1iKhVq9XBVwpgm41tsBqrLYXe2gZrYakx6tKQkyxDIndK+veIaEbEJUlnJP1cvmUB6Kas22Bhd1kC+9uSbrF9tW1LukPSxXzLAtBNWbfBwu66BnZEPCXpYUlPSzrX/j2nc64LQBdl3QYLu8s0SyQiPhERPxURhyLiIxHxw7wLA7C3sm6Dhd2x4wyQqLJug4XdEdhAwsq4DRZ2x2eJAEAiCGwASARDIsCQsToRl4vABoZoY3XixoKXjdWJkghtdEVgoxRSuWrda3ViEetFsRDYSF5KV62sTkQ/uOmI5KX0mRqsTkQ/CGwkL6WrVlYnoh8MiSB5kxMVNXYI52FftWYZR2d1IvpBYCN5c7PT28awpeFftfYyjs7qRFwuhkSQvGMzUzp1/LCmJiqypKmJik4dPzzUUExpHB3p4gobpTDqq9aUxtGRLq6wgQFg9geGgcAGBoDZHxgGhkSAAWD2B4aBwAYGZNTj6Cg/hkQAIBEENgAkomtg2562/cyWr1dtPziE2gAAW3Qdw46IZUk3S5LtMUkNSY/kWxYAoFOvQyJ3SPpWRLyYRzEAgN31Gtj3SXpopydsn7Bdt11vNpv9VwYA2CZzYNu+UtIHJf3jTs9HxOmIqEVErVqtDqo+AEBbL1fY75f0dET8V17FAAB210tg/6p2GQ4BAOQv00pH21dL+iVJv5VvOUD5pbJhMIonU2BHxOuSrs25FqD0UtowGMXDZ4mg1Ip2NbvXRgcENrohsFFaRbyaZaMD9IPPEkFpFXHbLjY6QD8IbJRWEa9m2egA/SCwUVpFvJotwobBSBdj2CitudnpbWPYUjGuZtnoAJeLwEZpsW0XyobARqlxNYsyYQwbABJBYANAIghsAEgEgQ0AiSCwASARBDYAJILABoBEENgAkAgCGwASQWADQCIIbABIBIENAInIFNi2J2w/bPs52xdtvy/vwgAA22X9tL4/l/TliPiQ7SslXZ1jTQCAHXQNbNvvknSbpF+XpIh4Q9Ib+ZYFAOiUZUjkPZKakv7G9pLtT9m+Jue6AAAdsgT2FZKOSPqriJiR9L+SPt7ZyPYJ23Xb9WazOeAyAQBZAvslSS9FxFPt7x/WeoBvExGnI6IWEbVqtTrIGgEAyhDYEfGfkr5je2Pn0jskfTPXqgAAb5N1lsjvSfpce4bIC5I+ml9JAICdZArsiHhGUi3fUgAAe2GlIwAkgsAGgERkHcMGtLDU0PzislZWW5qcqGhudlrHZqZGXRawbxDYyGRhqaGTZ86pdWlNktRYbenkmXOSVKjQ5k0FZcaQCDKZX1zeDOsNrUtrml9cHlFFb7fxptJYbSn01pvKwlJj1KUBA0FgI5OV1VZPx0chhTcVoB8ENjKZnKj0dHwUUnhTAfpBYCOTudlpVcbHth2rjI9pbnZ6l98xfCm8qQD9ILCRybGZKZ06flhTExVZ0tRERaeOHy7UDb0U3lSAfjBLBJkdm5kqVEB32qiNWSIoKwIb26Q+La7obypAPwhsbEplrjWwXzGGjU1MiwOKjcDGJqbFAcVGYGMT0+KAYiOwsYlpcUCxcdMRm5gWBxQbgY1tmBYHFBdDIgCQCAIbABJBYANAIjKNYdv+D0k/kLQm6c2IYAd1DEzqy+GBYenlpuMvRMQruVWCfYnl8EB2DIlgpFgOD2SXNbBD0uO2z9o+sVMD2yds123Xm83m4CpEqbEcHsgua2DfGhFHJL1f0v22b+tsEBGnI6IWEbVqtTrQIlFeLIcHsssU2BGx0v71ZUmPSDqaZ1HYP1gOD2TXNbBtX2P7nRuPJf2ypPN5F4b9IYWtx4CiyDJL5MclPWJ7o/3fR8SXc60K+wrL4YFsugZ2RLwg6aYh1AIA2APT+gAgEQQ2ACSCwAaARBDYAJAIAhsAEkFgA0AiCGwASASBDQCJILABIBEENgAkgsAGgEQQ2ACQCAIbABJBYANAIghsAEgEgQ0AiSCwASARBDYAJILABoBEZNmEdygWlhqaX1zWympLkxMVzc1OszErAGyRObBtj0mqS2pExD2DLGJhqaGTZ86pdWlNktRYbenkmXOSRGgDQFsvQyIPSLqYRxHzi8ubYb2hdWlN84vLefw4AEhSpsC2fb2kD0j6VB5FrKy2ejoOAPtR1ivsP5P0h5L+b7cGtk/YrtuuN5vNnoqYnKj0dBwA9qOugW37HkkvR8TZvdpFxOmIqEVErVqt9lTE3Oy0KuNj245Vxsc0Nzvd0+sAQJlluel4q6QP2r5b0lWS3mX77yLiw4MqYuPGIrNEAGB3jojsje3bJf1Bt1kitVot6vV6f5UBwD5i+2xE1PZqw8IZAEhETwtnIuIJSU/kUgkAYE9cYQNAIghsAEgEgQ0AiSCwASARBDYAJILABoBEENgAkAgCGwASQWADQCIIbABIBIENAIkgsAEgEQQ2ACSCwAaARBDYAJAIAhsAEkFgA0AiCGwASASBDQCJILABIBFdA9v2Vba/bvtZ2xdsf3IYhQEAtsuya/oPJf1iRLxme1zS12w/FhFP5lwbAGCLroEdESHptfa34+2vyLMoAMDbZRrDtj1m+xlJL0v6SkQ8lWtVAIC3yRTYEbEWETdLul7SUduHOtvYPmG7brvebDYHXCYAoKdZIhGxKukJSXft8NzpiKhFRK1arQ6mOgDApiyzRKq2J9qPK5LulPRcznUBADpkmSVynaTP2h7TesB/PiIezbcsAECnLLNEviFpZgi1AAD2kOUKe99YWGpofnFZK6stTU5UNDc7rWMzU6MuCwAkEdibFpYaOnnmnFqX1iRJjdWWTp45J0mENoBC4LNE2uYXlzfDekPr0prmF5dHVBEAbEdgt62stno6DgDDRmC3TU5UejoOAMNGYLfNzU6rMj627VhlfExzs9MjqggAtuOmY9vGjUVmiQAoKgJ7i2MzUwQ0gMJiSAQAEkFgA0AiCGwASASBDQCJILABIBFe37JxwC9qNyW9OPAXvnwHJL0y6iL6kHr9EudQFKmfQ+r1S7ufw09GxJ67v+QS2EVjux4RtVHXcblSr1/iHIoi9XNIvX6pv3NgSAQAEkFgA0Ai9ktgnx51AX1KvX6JcyiK1M8h9fqlPs5hX4xhA0AZ7JcrbABIHoENAIkoTWDbvsv2su3nbX98h+dvt/1928+0v/54FHXuxfZnbL9s+/wuz9v2X7TP8Ru2jwy7xr1kqD+FPrjB9j/Zvmj7gu0HdmhT2H7IWH+h+8H2Vba/bvvZ9jl8coc2he0DKfM59N4PEZH8l6QxSd+S9B5JV0p6VtLPdLS5XdKjo661y3ncJumIpPO7PH+3pMckWdItkp4adc091p9CH1wn6Uj78Tsl/esOf5cK2w8Z6y90P7T/XN/Rfjwu6SlJt6TSBz2cQ8/9UJYr7KOSno+IFyLiDUn/IOneEdfUs4j4qqTv7dHkXkl/G+uelDRh+7rhVNddhvoLLyK+GxFPtx//QNJFSZ0fkl7YfshYf6G1/1xfa3873v7qnB1R2D6QMp9Dz8oS2FOSvrPl+5e081/S97X/i/KY7Z8dTmkDlfU8iyyZPrB9UNKM1q+OtkqiH/aoXyp4P9ges/2MpJclfSUikuuDDOcg9dgPZQls73Cs893saa2v1b9J0l9KWsi7qBxkOc8iS6YPbL9D0hckPRgRr3Y+vcNvKVQ/dKm/8P0QEWsRcbOk6yUdtX2oo0nh+yDDOfTcD2UJ7Jck3bDl++slrWxtEBGvbvwXJSK+JGnc9oHhlTgQXc+zyFLpA9vjWg+7z0XEmR2aFLofutWfSj9IUkSsSnpC0l0dTxW6D7ba7Rwupx/KEtj/Ium9tt9t+0pJ90n64tYGtn/CttuPj2r93P976JX254uSfq19h/wWSd+PiO+OuqisUuiDdn2flnQxIv50l2aF7Ycs9Re9H2xXbU+0H1ck3SnpuY5mhe0DKds5XE4/lGIT3oh40/bvSlrU+oyRz0TEBdu/3X7+ryV9SNLv2H5TUkvSfdG+VVsUth/S+p3jA7ZfkvQJrd+s2DiHL2n97vjzkl6X9NHRVLqzDPUXvg8k3SrpI5LOtccfJemPJN0oJdEPWeovej9cJ+mztse0HmKfj4hHO/49F7kPpGzn0HM/sDQdABJRliERACg9AhsAEkFgA0AiCGwASASBDQCJILABIBEENgAk4v8BcE8sn0KhT7EAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 来看看产生的x-y分布\n",
    "x, y = get_fake_data(batch_size=16)\n",
    "plt.scatter(x.squeeze().cpu().numpy(), y.squeeze().cpu().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:04:40.355354Z",
     "start_time": "2020-12-09T03:04:34.303228Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbOUlEQVR4nO3deZSU9b3n8feXxdiurYIIjS0qDYo00NgxCzdGAcUtLrgkmuRmcpNhzsxVce6M9+qZM5ObmcmVK7kGXBPcjY6J92pMbjYXVnFNoygqIjvSoODCJr1WfeePqoKmqKarq56qp+qpz+scjnR1ddev6iRvfuf3/J7nMXdHRETKX5+wByAiIsFQ0EVEIkJBFxGJCAVdRCQiFHQRkYjoV8wXGzBggA8bNqyYLykiUlZ2tXby0c5WdrfH6N+3DwMP/wLr3lv2sbsP7Olnixr0YcOG0dTUVMyXFBEpee7OCys/Ztbz7/P6hm2ceuTB/Jezh3Nl41C+0K8vZrY+m99T1KCLiMhe7s6iZMjf2LCNIUcezE8uG80VpydC3lsKuohIkQUd8hQFXUSkSNydhe9vZdbzK1n6wTZqqqv4p8vqueL0oRzUL/89Kgq6iEiBFTrkKQq6iEiBuDsLkiF/MxnyW6bWc/n4YEOeoqCLSEE8/UYzM59ZwaZtLQypruLGKSO5tKEm7GEVRbFDnqKgi0jgnn6jmZufWkZLRwyA5m0t3PzUMoBIR93dWbBiK7Oef583N24vWshTFHQRCdzMZ1bsiXlKS0eMmc+siGTQM4V8xtR6phYp5CkKuogEbtO2ll49Xq7cnfkrtjDr+ZW8tXE7Q48KJ+QpCrqIBG5IdRXNGeI9pLoqhNEEL1PI//nyRMj79w3vElkKuogE7sYpI/dZQweo6t+XG6eMDHFU+XN35r23hdlzSyvkKQq6iAQutU4elV0uqZDPen4ly5q3c/zRVdx6+RguG19TEiFP6THoZvYAcBGwxd1HJx+bCXwDaAdWA993920FHKeIlJlLG2rKNuAp5RLylGxm6A8BdwKPdHnsOeBmd+80s38Gbgb+IfjhiYgUn7szd3liaWVZ83Zqjz6EW68Yw2UNpRnylB6D7u6LzGxY2mPPdvnyFeCKgMclIlL0k5NSIZ81933ebt5RNiFPCWIN/W+AX3f3TTObBkwDqK2tDeDlRKQSFOrkpEz/SFwybgjPL9/C7C4hn3nFGC4tk5CnmLv3/KTEDP33qTX0Lo//D6ARmOpZ/KLGxkbXDS5EJBsTZszLuPWxprqKF2+amNPvTP9HAuCgvn049vAvsHFbC7VHH8J1E4eXXMjNbIm7N/b0vJxn6Gb2PRIHSydlE3MRkd4oxMlJmc5gbY/F2byjtSxn5OlyGrmZnUfiIOjF7r472CGJiHR/ElI+JydlmvEDxOLOlY3Hl3XMIYugm9njwMvASDPbaGY/ILHr5XDgOTNbamY/L/A4RaTC3DhlJFX99717T64nJ7k7z7zzIf37Wsbv10TkDNZsdrlcneHh+wswFhGRPYI4Ocndefbdj5j9/Ere3byDAYcdxPaWDjpie1eJo3AGa4rOFBWRkpXryUnxeDLkc1eyfPMOhh1zCP9y5VguGTeE37+1OTJnsKZT0EUkMtJDfuKAQ7ntqrFcPHYI/ZLr41E4g7U7CrqIlL1EyD9k1vMree/DnRlDXgkUdBEpW5lC/rNvjuUbYyor5CkKuoiUnXg8sWtl9txEyE+q8JCnKOgiUjYU8gNT0EWk5GUK+axvjuMbY4fQt0/mveUpxb7AV5gUdBEpWfG48+d3PmT28ytZ8VHvQg6Fu8BXqVLQRaTk7BfygYcy+1vjuGhMdiFPyXTtlpaOGDOfWaGgi4gUUjzu/OntD7l9bn4hTynEBb5KmYIuIvsIY805FfLZc9/n/Y92cXKeIU8ZUl2V8YJc+Vzgq5Qp6CKyR7HXnONx549vb+b2uSsDDXnKjVNG7nf98yhduyWdgi4iexRrzTk95MOPPYzbr27gwvrBgYQ8JYgLfJUTBV1E9ij0mnMs7vxxWSLkK7cULuRdpUd95jMr9nk8ShR0kRJW7PXsQq05Zwr5HVc3cEE3IQ/yfVfS1kUFXaREhRGioNecextyCP59V9LWRQVdpESFEaKg1pxjcecPyZCv2rKLuixCnhL0+66krYsKukiJCitE+VwvPFPI77ymgQtGD6ZPlmvkQb/vStq6qKCLlKhyClEs7vz+rU3cPnclq7d+nlPIU4J+35W0dVGXJxMpUUHeJLlQYnHnt0ubOfdnC5n+q6X07WPcdc14nrnhTC4aM6TXMYfg3/elDTXcMrWemuoqjMQNoW+ZWh+59XPQDF2kZJXyHur0GfmIQYdx1zXjOX/0cTlFvKtCvO8o33auK3P3np8VkMbGRm9qaira64lIsFIhnz13JWu2fs7IQYczfXId552Wf8ile2a2xN0be3qeZugiEVKofeuZQn73t8cr5CVGQReJiELsW4/FnX9/cxO3z0uE/JTjDueeb49nSoaQV9KNJEpVj0E3sweAi4At7j46+djRwK+BYcA64Cp3/6xwwxSRngS5f7s3IYfKOhuzlGWzy+Uh4Ly0x24C5rp7HTA3+bWIhCiI/dudsTi/eWMj59y2kBt+vZSD+vbhnm+P54/Xf43z67vfgnigf0ykeHqcobv7IjMblvbwJcBZyb8/DCwA/iHIgYlI7+Szf7szFuff39rEHXNXsebjxIz8598Zz7mjslsjr6SzMUtZrmvog9x9M4C7bzazYwMck4jkIJcTaDpjcX735ibumLeKtTmEPKWcToKKsoIfFDWzacA0gNra2kK/nEjF6s3+7cwhP51zRw3K+WSgSjkbs5TlGvSPzGxwcnY+GNjS3RPdfQ4wBxL70HN8PRHJQk8n0HTG4vx26SbumLeSdZ/s5tTBR+QV8q6vC6V5ElQlyTXovwO+B8xI/ve3gY1IRPYIaitgppD/4runc86p+YW8q0o5G7OUZbNt8XESB0AHmNlG4EckQv6Emf0A2ABcWchBilSiILYCpod8VAFCLqUjm10uV3fzrUkBj0UkMFE4ySWffeWdsThPL93EnV1CPue7p3POqEGYKeRRpTNFJXKicpJLLlsBUyG/Y95K1ivkFUdBl8iJyi3HerMVMHFCUDN3zl+lkFcwBV0iJyonuWSzFTA95KcNOYJ7/7qRyaceq5BXIAVdIicqJ7kcaCtgRyrk81ax4VOFXBIUdImcKJ3kkr4VsCMW54mmD/aEfHTNEdz3141MUsgFBV0iKIonuaTPyBVyyURBl0gqhZNcgtg62RGL85vXm7lj/ko++LSF+pojuf97jUw8RSGX/SnoIgWQ79bJTCH/x++dppDLASnoIgWQ69bJjlicp17fyJ3zV/HBpy2MGXokP774NM4eqZBLzxR0kQLo7dbJVMjvmLeKjZ8p5JIbBV2kALLdOtkRi/PkksSMPBXy/32JQi65UdBF8tDdgc+etk62d+5dWtn4WQtjhx7J/7lkNGeNHKiQS84UdJEcZXPgMz32F9QP5vHXNnDnvFU0b1PIJVjmXrx7TjQ2NnpTU1PRXk+kkCbMmJdxWaWmuooXb5q4z2PtnXGefH3j3pAfX80Nk+s4a4RCLj0zsyXu3tjT8zRDF8lRNgc+M4X8/142WiGXglDQRXJ0oAOf7Z1x/m3JRu6anwj5uOOr+cllo/m6Qi4FpKCL5CjTgc+D+/VhwvBjOPunCxRyKToFXSRHXQ98Nm9robqqP2bwRNNGxh1fzT9NrefMugEKuRSNgi6ShwvqB/N5eyd3z19N87YWGmqrmTV5hEIuoVDQRXLQ1hnjX5s2cvf8VWza3kpDbTW3TK3nawq5hEhBF+mF9JCPr61mxuVjFHIpCQq6SBbaOmM8kQz5ZoVcSpSCLnIAmUJ+6xVj+KvhCrmUHgVdJIP0kJ9+wlG9DnkQN7gQ6Q0FXaSLts4YT/zlA+5esHpPyGdeMZYJw4/p1Yw83xtciOQir6Cb2X8Ffgg4sAz4vru3BjEwiY5ymKmmh7yxFyHP9P5yvcGFSD5yDrqZ1QDXA6PcvcXMngC+BTwU0NgkAkp9ptraEeOJpg+4e/5qPtyRCPlPrxzLV0/Obkbe3ftLj3lKd9d/EQlCvksu/YAqM+sADgE25T8kiZJSnammh/yLw47iX67KPuQp3b2/vmbEMlzJNP0GFyJByjno7t5sZj8FNgAtwLPu/mz688xsGjANoLa2NteXkzLV21uxFVprR4xf/+UD7l6wio92tPHFYUdx21Vj+UovQ57S3fuIuVPVv2+3N7gQKYQ+uf6gmR0FXAKcCAwBDjWz76Q/z93nuHujuzcOHDgw95FKWepuRlrsmWprR4yHX1rH12fO50e/e4cTjj6U//fDL/HEf/oKX81jC2J376OmuopbptZTU12Fdfm6FJaZJLryWXKZDKx1960AZvYU8FXg0SAGJtHQ063YCq21I8avXtvAPQtX89GONs4YdjQ/u2pczjPydAd6f5c21CjgUlT5BH0D8GUzO4TEksskQLcjkn10dyu2QocuY8i/OY6vnJR7yA+0W6fUd/FIZcjrFnRm9mPgm0An8AbwQ3dv6+75ugWdFFoq5HcvWM2WnW2cceLR3DC5Lq+Qw/67WSAxE9cyihRDUW5B5+4/An6Uz+8QCUJrR4zHX9vAPV1CPvtbDXzl5GMC+f2lultHpCudKSplI9OSx3mjj9sn5F8KOOQppbZbRyQTBV3KQqYTeG781zf5X799mx2tnXz5pMKEPOVA9w8VKRUKupSFTEseHXHHOuM8/h+/XLCQp4S9W0ckGwq6lIVMs2OAjs54wWMO4e3WEekNBV1KWmtHjMde3UAfg3iGDVnFXPLQvnIpdQq6lKSW9hiPvbqeny9cw8e72qg79jDWf7Kb9lh8z3O05CGyLwVdSkp6yL968jHcdU0DXzrpmLK4DK9ImBR0KQkHCnmKljxEDkxBl1DtDflqPt7VzoThx3D3pPGcceLRYQ9NpOwo6BKKzCEfoZCL5EFBl6La3d7JY69s4BeLEiH/q+EDmD65ji8OU8hF8qWgS1Eo5CKFp6BLQe1u7+TRV9bzi4Vr+OTzdr5WN4Dpk+poVMhFAqegS0Eo5CLFp6BLoHa3d/LLl9czZ9HekN8wuY7TT1DIRQpNQZdAKOQi4VPQJS+ft3Xyy1cSIf9UIRcJlYIuOUkP+ZkjBjJ9Uh2nn3BU2EMTqVgKuvSKQi5SuhR0ycrnbZ088vJ67n0hEfKvjxjI9Ml1jK/tfch1kS2RwlDQ5YBSIZ+zaDWf7e7IK+SQ+VZyNz+1DEBRF8mTgi4Z7Wrr5JGX13HvojV8truDs0YmllYacgx5SqZbybV0xJj5zAoFXSRPCrrso1AhT9nUza3kuntcRLKnoAuQCPnDL63j3hfWsK0AIU8ZUl2V8f6gxbyVnEhU5RV0M6sG7gNGAw78jbu/HMC4pEjSQ372yIFMnzyCccdXF+T1bpwycp81dNCt5ESCku8MfTbwZ3e/wswOAg4JYExSBDtbO/bsWilGyLs6uH+fPUGvrurPP158mtbPRQKQc9DN7AjgTOA/ALh7O9AezLCkUNJDPvGUY7l+Ul1RQp6+wwWgrTN+gJ8Qkd7IZ4Z+ErAVeNDMxgJLgOnu/nnXJ5nZNGAaQG1tbR4vJ/nY2dqRXFpZy/aWRMinT6pjbBFCnqIdLiKFlU/Q+wHjgevc/VUzmw3cBPzPrk9y9znAHIDGxkbP4/UkB+khn5SckRcz5Cna4SJSWPkEfSOw0d1fTX79bySCLiUgU8inT65jzNDq0MakHS4ihZVz0N39QzP7wMxGuvsKYBLwbnBDk1zsbO3goRfXcd/i0gl5ina4iBRWvrtcrgMeS+5wWQN8P/8hSS52tHbwcJeQTz41sbRSCiFPSa2T6zouIoWRV9DdfSnQGMxQJBc7UjPyF9awo7WTyacey/RJI6gfemTYQ8vo0oYaBVykQHSmaJnaP+SDmD6prmRDLiKFp6CXmR2tHTy4eB33L1bIRWRfCnqZ2N6SmJGnQn7OqETIR9co5CKSoKCXuO0tHTz44lruX7yWnQq5iByAgl6iFHIR6S0FvcSkh/zcUYO4XiEXkSwo6CVie0sHDyxeywMvKuQikhsFPWTpIZ9yWiLkpw1RyEWkdxT0kGzf3cH9L67lwcVr2dmmkItI/hT0IksP+XmnHcf1k+oYNeSIsIcmImVOQS+S7bs7uH/xGh58cd2eGfn0SSMUchEJjILejaffaA7kIlLbdrfzwOK1e0J+/ujEjPzUwQq5iARLQc8g/VZpzdtauPmpZQBZR33b7nbuT4Z8l0IuIkWgoGeQz63SFHIRCYuCnkF3t0Rr3tbChBnzMi6/pIf8gvrjuG6iQi4ixaOgZ9DdrdJg/+WXbbvbue+FtTz00t6QXz+pjlOOU8hFpLgUdPY/AHr2KQN5cknzfssuKS0dMWb86T1Wbdm1J+QX1g/muknDFXIRCY25e9FerLGx0Zuamor2etlIPwAKiftcXn56DfPf29rtTD1FIReRQjOzJe7e493hKn6G3t0B0PnvbeXFmyYyYca8jFGv6t+Xp/92AiOPO7xYQxUROaA+YQ8gbN0dAE09/p/POpl+fWyf732hXx9umVqvmItISan4GXp3B0AHHXEwt/75PR5+aR2xuFPVvy8tHTFqdKd6ESlRFR/0G6eM3G8NvV8f47Pd7dyzcDUX1g/m+kl1jBik2biIlLaKD3pqpj3jT+/x4Y5WDIjFnfPrB3P9xOHUFTHkQV1uQEQqU8UH/dPP21nx0U52tHZgBheNGVL0kEMwlxsQkcpWsUH/ZFcb976wlkdeXkdLR4xvjBnCdSGEPCWfyw2IiEAAQTezvkAT0OzuF+U/pMLKFPLrJw1n+LHhrpH3tNtGRKQnQczQpwPLgZI+s+aTXW3MeWENv3x5fUmFPKW73TZDqqtCGI2IlKO8gm5mQ4ELgZ8AfxfIiLoI4iBhKuSPvLSe1s4YF49NLK2USshTMu22qerflxunjAxxVCJSTvKdoc8C/h7oto5mNg2YBlBbW5v1L873IOHHu9q4d9EaHnm5tEOeknpP2uUiIrnKOehmdhGwxd2XmNlZ3T3P3ecAcyBxLZdsf3+uBwm7hrwtGfJrJ9Yx/NjDenzNsLcNXtpQo4CLSM7ymaFPAC42swuAg4EjzOxRd/9OEAPr7UHCfEIO2jYoIuUv56C7+83AzQDJGfp/DyrmkP1Bwo93tTFnUeJgZ1tnjEvG1XDtxOGcPDC7kKdo26CIlLuS3Yfe00HCoEKeUsxtg2Ev7YhINAUSdHdfACwI4neldHeQcMLwAfzkD+/yy1fW094Z59JkyE/KMeQpxdo2qKUdESmUkp2hw74HCbfubGPOotXc9NRbgYY8pVjbBrW0IyKFUtJBB9iys5U5C9fw6KvJGXlDDdeeHVzIU4q1bVBnhIpIoZRs0DOF/LqJdZw44NCCvWYxtg3qjFARKZSSC3p6yC9rGMq1E4cXNOTFpDNCRaRQSiboW3a28ouFa3j0lfV0xn3PGnlvQ17qO0h0RqiIFEroQc8U8usmDmdYDjPyctlBojNCRaQQQgv6lh2t/HzhGh57NRHyy5IHO3MJeUoQO0hKfYYvItKdoge9ECFPyXcHSbnM8EVEMilq0Ddvb+Vrt86nM+5MbUiskZ9wTHAHO/PdQaI94iJSzooa9I93tXHN2CGBhzwl3x0k2iMuIuWsqEEfOehwZl45tmC/P98dJNojLiLlrKhBP6hfn4K/Rj47SLRHXETKWejbFkuJ9oiLSDlT0NNoj7iIlKvCr4GIiEhRKOgiIhGhoIuIRISCLiISEQq6iEhEKOgiIhGhoIuIRISCLiISEQq6iEhEKOgiIhGRc9DN7Hgzm29my83sHTObHuTARESkd/K5lksn8N/c/XUzOxxYYmbPufu7AY1NRER6IecZurtvdvfXk3/fCSwHdFUrEZGQBLKGbmbDgAbg1Qzfm2ZmTWbWtHXr1iBeTkREMsg76GZ2GPAkcIO770j/vrvPcfdGd28cOHBgvi8nIiLdyCvoZtafRMwfc/enghmSiIjkIp9dLgbcDyx399uCG5KIiOQinxn6BOC7wEQzW5r8c0FA4xIRkV7Keduiuy8GLMCxiIhIHnSmqIhIRCjoIiIRoaCLiESEgi4iEhEKuohIRCjoIiIRoaCLiESEgi4iEhEKuohIRCjoIiIRoaCLiESEgi4iEhEKuohIRCjoIiIRoaCLiESEgi4iEhEKuohIRCjoIiIRoaCLiESEgi4iEhEKuohIRCjoIiIRoaCLiESEgi4iEhEKuohIROQVdDM7z8xWmNkqM7spqEGJiEjv5Rx0M+sL3AWcD4wCrjazUUENTEREeiefGfoZwCp3X+Pu7cCvgEuCGZaIiPRWvzx+tgb4oMvXG4EvpT/JzKYB05JftpnZ23m8ZpQMAD4OexAlQp/FXvos9tJnsdfIbJ6UT9Atw2O+3wPuc4A5AGbW5O6NebxmZOiz2EufxV76LPbSZ7GXmTVl87x8llw2Asd3+XoosCmP3yciInnIJ+h/AerM7EQzOwj4FvC7YIYlIiK9lfOSi7t3mtm1wDNAX+ABd3+nhx+bk+vrRZA+i730Weylz2IvfRZ7ZfVZmPt+y94iIlKGdKaoiEhEKOgiIhFRlKDrEgF7mdkDZral0vfjm9nxZjbfzJab2TtmNj3sMYXFzA42s9fM7M3kZ/HjsMcUNjPra2ZvmNnvwx5LmMxsnZktM7Ol2WxdLPgaevISAe8D55DY6vgX4Gp3f7egL1yizOxMYBfwiLuPDns8YTGzwcBgd3/dzA4HlgCXVuL/LszMgEPdfZeZ9QcWA9Pd/ZWQhxYaM/s7oBE4wt0vCns8YTGzdUCju2d1glUxZui6REAX7r4I+DTscYTN3Te7++vJv+8ElpM4+7jieMKu5Jf9k38qdreCmQ0FLgTuC3ss5aYYQc90iYCK/D+uZGZmw4AG4NWQhxKa5BLDUmAL8Jy7V+xnAcwC/h6IhzyOUuDAs2a2JHkZlQMqRtCzukSAVCYzOwx4ErjB3XeEPZ6wuHvM3ceROOP6DDOryOU4M7sI2OLuS8IeS4mY4O7jSVzV9m+TS7bdKkbQdYkAySi5Xvwk8Ji7PxX2eEqBu28DFgDnhTuS0EwALk6uHf8KmGhmj4Y7pPC4+6bkf7cAvyGxhN2tYgRdlwiQ/SQPBN4PLHf328IeT5jMbKCZVSf/XgVMBt4LdVAhcfeb3X2ouw8j0Yp57v6dkIcVCjM7NLlhADM7FDgXOODuuIIH3d07gdQlApYDT2RxiYDIMrPHgZeBkWa20cx+EPaYQjIB+C6JGdjS5J8Lwh5USAYD883sLRIToOfcvaK36wkAg4DFZvYm8BrwB3f/84F+QKf+i4hEhM4UFRGJCAVdRCQiFHQRkYhQ0EVEIkJBFxGJCAVdRCQiFHQRkYj4/zUJDp2D5J++AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "w:  2.0206243991851807 b:  2.942985773086548\n"
     ]
    }
   ],
   "source": [
    "# 随机初始化参数\n",
    "w = t.rand(1, 1).to(device)\n",
    "b = t.zeros(1, 1).to(device)\n",
    "\n",
    "lr =0.02 # 学习率\n",
    "\n",
    "for ii in range(500):\n",
    "    x, y = get_fake_data(batch_size=4)\n",
    "    \n",
    "    # forward：计算loss\n",
    "    y_pred = x.mm(w) + b.expand_as(y) # x@W等价于x.mm(w);for python3 only\n",
    "    '''\n",
    "    torch.mul(a, b)是矩阵a和b对应位相乘，a和b的维度必须相等，比如a的维度是(1, 2)，b的维度是(1, 2)，返回的仍是(1, 2)的矩阵\n",
    "    torch.mm(a, b)是矩阵a和b矩阵相乘，比如a的维度是(1, 2)，b的维度是(2, 3)，返回的就是(1, 3)的矩阵\n",
    "    '''\n",
    "    loss = 0.5 * (y_pred - y) ** 2 # 均方误差\n",
    "    loss = loss.mean()\n",
    "    \n",
    "    # backward：手动计算梯度\n",
    "    dloss = 1\n",
    "    dy_pred = dloss * (y_pred - y)\n",
    "    \n",
    "    dw = x.t().mm(dy_pred)\n",
    "    db = dy_pred.sum()\n",
    "    \n",
    "    # 更新参数\n",
    "    w.sub_(lr * dw)\n",
    "    b.sub_(lr * db)\n",
    "    \n",
    "    if ii%50 ==0:\n",
    "       \n",
    "        # 画图\n",
    "        display.clear_output(wait=True)\n",
    "        x = t.arange(0, 6).view(-1, 1)\n",
    "        y = x.float().mm(w) + b.expand_as(x)\n",
    "        plt.plot(x.cpu().numpy(), y.cpu().numpy()) # predicted\n",
    "        \n",
    "        x2, y2 = get_fake_data(batch_size=32) \n",
    "        plt.scatter(x2.numpy(), y2.numpy()) # true data\n",
    "        \n",
    "        plt.xlim(0, 5)\n",
    "        plt.ylim(0, 13)\n",
    "        plt.show()\n",
    "        plt.pause(0.5)\n",
    "        \n",
    "print('w: ', w.item(), 'b: ', b.item())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可见程序已经基本学出w=2、b=3，并且图中直线和数据已经实现较好的拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "虽然上面提到了许多操作，但是只要掌握了这个例子基本上就可以了，其他的知识，读者日后遇到的时候，可以再看看这部份的内容或者查找对应文档。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 autograd\n",
    "\n",
    "用Tensor训练网络很方便，但从上一小节最后的线性回归例子来看，反向传播过程需要手动实现。这对于像线性回归等较为简单的模型来说，还可以应付，但实际使用中经常出现非常复杂的网络结构，此时如果手动实现反向传播，不仅费时费力，而且容易出错，难以检查。torch.autograd就是为方便用户使用，而专门开发的一套自动求导引擎，它能够根据输入和前向传播过程自动构建计算图，并执行反向传播。\n",
    "\n",
    "计算图(Computation Graph)是现代深度学习框架如PyTorch和TensorFlow等的核心，其为高效自动求导算法——反向传播(Back Propogation)提供了理论支持，了解计算图在实际写程序过程中会有极大的帮助。本节将涉及一些基础的计算图知识，但并不要求读者事先对此有深入的了解。关于计算图的基础知识推荐阅读Christopher Olah的文章[^1]。\n",
    "\n",
    "[^1]: http://colah.github.io/posts/2015-08-Backprop/\n",
    "\n",
    "\n",
    "### 3.2.1 requires_grad\n",
    "PyTorch在autograd模块中实现了计算图的相关功能，autograd中的核心数据结构是Variable。从v0.4版本起，Variable和Tensor合并。我们可以认为需要求导(requires_grad)的tensor即Variable. autograd记录对tensor的操作记录用来构建计算图。\n",
    "\n",
    "Variable提供了大部分tensor支持的函数，但其不支持部分`inplace`函数，因这些函数会修改tensor自身，而在反向传播中，variable需要缓存原来的tensor来计算反向传播梯度。如果想要计算各个Variable的梯度，只需调用根节点variable的`backward`方法，autograd会自动沿着计算图反向传播，计算每一个叶子节点的梯度。\n",
    "\n",
    "`variable.backward(gradient=None, retain_graph=None, create_graph=None)`主要有如下参数：\n",
    "\n",
    "- grad_variables：形状与variable一致，对于`y.backward()`，grad_variables相当于链式法则${dz \\over dx}={dz \\over dy} \\times {dy \\over dx}$中的$\\textbf {dz} \\over \\textbf {dy}$。grad_variables也可以是tensor或序列。\n",
    "- retain_graph：反向传播需要缓存一些中间结果，反向传播之后，这些缓存就被清空，可通过指定这个参数不清空缓存，用来多次反向传播。\n",
    "- create_graph：对反向传播过程再次构建计算图，可通过`backward of backward`实现求高阶导数。\n",
    "\n",
    "上述描述可能比较抽象，如果没有看懂，不用着急，会在本节后半部分详细介绍，下面先看几个例子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:12:43.886874Z",
     "start_time": "2020-12-09T03:12:43.883905Z"
    }
   },
   "outputs": [],
   "source": [
    "import torch as t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:13:04.382924Z",
     "start_time": "2020-12-09T03:13:04.377906Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.2514412403,  0.2462205738, -1.0326300859,  0.1911817789],\n",
       "        [-0.2160580903,  0.1762175262,  0.5185296535,  0.7876414657],\n",
       "        [-0.7803510427, -1.0841886997,  2.1908969879,  1.1424160004]],\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#在创建tensor的时候指定requires_grad\n",
    "a = t.randn(3,4, requires_grad=True)\n",
    "# 或者\n",
    "a = t.randn(3,4).requires_grad_()\n",
    "# 或者\n",
    "a = t.randn(3,4)\n",
    "a.requires_grad=True\n",
    "a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:13:21.532883Z",
     "start_time": "2020-12-09T03:13:21.528894Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0.],\n",
       "        [0., 0., 0., 0.]], requires_grad=True)"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = t.zeros(3,4).requires_grad_()\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:13:32.843864Z",
     "start_time": "2020-12-09T03:13:32.839845Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 1.2514412403,  0.2462205738, -1.0326300859,  0.1911817789],\n",
       "        [-0.2160580903,  0.1762175262,  0.5185296535,  0.7876414657],\n",
       "        [-0.7803510427, -1.0841886997,  2.1908969879,  1.1424160004]],\n",
       "       grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 也可写成c = a + b\n",
    "c = a.add(b)\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:13:42.351783Z",
     "start_time": "2020-12-09T03:13:42.343777Z"
    }
   },
   "outputs": [],
   "source": [
    "d = c.sum()\n",
    "d.backward() # 反向传播"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:13:53.068322Z",
     "start_time": "2020-12-09T03:13:53.064333Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d # d还是一个requires_grad=True的tensor,对它的操作需要慎重\n",
    "d.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:14:00.589559Z",
     "start_time": "2020-12-09T03:14:00.585570Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]])"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:15:17.823821Z",
     "start_time": "2020-12-09T03:15:17.819802Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "execution_count": 149,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 此处虽然没有指定c需要求导，但c依赖于a，而a需要求导，\n",
    "# 因此c的requires_grad属性会自动设为True\n",
    "a.requires_grad, b.requires_grad, c.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:15:32.252984Z",
     "start_time": "2020-12-09T03:15:32.248965Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, False)"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 由用户创建的variable属于叶子节点，对应的grad_fn是None\n",
    "a.is_leaf, b.is_leaf, c.is_leaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:16:08.636764Z",
     "start_time": "2020-12-09T03:16:08.632803Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 151,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# c.grad是None, 因c不是叶子节点，它的梯度是用来计算a的梯度\n",
    "# 所以虽然c.requires_grad = True,但其梯度计算完之后即被释放\n",
    "c.grad is None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算下面这个函数的导函数：\n",
    "$$\n",
    "y = x^2\\bullet e^x\n",
    "$$\n",
    "它的导函数是：\n",
    "$$\n",
    "{dy \\over dx} = 2x\\bullet e^x + x^2 \\bullet e^x\n",
    "$$\n",
    "来看看autograd的计算结果与手动求导计算结果的误差。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:16:42.955236Z",
     "start_time": "2020-12-09T03:16:42.951245Z"
    }
   },
   "outputs": [],
   "source": [
    "def f(x):\n",
    "    '''计算y'''\n",
    "    y = x**2 * t.exp(x)\n",
    "    return y\n",
    "\n",
    "def gradf(x):\n",
    "    '''手动求导函数'''\n",
    "    dx = 2*x*t.exp(x) + x**2*t.exp(x)\n",
    "    return dx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:16:49.939202Z",
     "start_time": "2020-12-09T03:16:49.928202Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[5.2578204870e-01, 5.4110878706e-01, 3.8257021904e+00, 4.2048674822e-01],\n",
       "        [5.6707000732e+01, 4.1849277914e-02, 5.4113256931e-01, 4.5228581429e+00],\n",
       "        [2.3195090294e+00, 2.5981354993e-03, 2.3006124794e-01, 5.9620153904e-01]],\n",
       "       grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.randn(3,4, requires_grad = True)\n",
    "y = f(x)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:16:58.332207Z",
     "start_time": "2020-12-09T03:16:58.327188Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 8.0440729856e-02,  1.1058032513e-02,  1.0668087959e+01,\n",
       "         -3.0723500252e-01],\n",
       "        [ 1.0520138550e+02, -3.2294842601e-01,  1.0485112667e-02,\n",
       "          1.2193264961e+01],\n",
       "        [ 7.2127270699e+00,  1.0710806400e-01, -4.5589101315e-01,\n",
       "          2.6582336426e+00]], grad_fn=<AddBackward0>)"
      ]
     },
     "execution_count": 154,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# autograd的计算结果与利用公式手动计算的结果一致\n",
    "gradf(x) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:17:09.760130Z",
     "start_time": "2020-12-09T03:17:09.752118Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 8.0440729856e-02,  1.1058032513e-02,  1.0668087959e+01,\n",
       "         -3.0723500252e-01],\n",
       "        [ 1.0520138550e+02, -3.2294842601e-01,  1.0485112667e-02,\n",
       "          1.2193264961e+01],\n",
       "        [ 7.2127270699e+00,  1.0710806400e-01, -4.5589101315e-01,\n",
       "          2.6582336426e+00]])"
      ]
     },
     "execution_count": 155,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.backward(t.ones(y.size())) # gradient形状与y一致\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2.2 计算图\n",
    "\n",
    "PyTorch中`autograd`的底层采用了计算图，计算图是一种特殊的有向无环图（DAG），用于记录算子与变量之间的关系。一般用矩形表示算子，椭圆形表示变量。如表达式$ \\textbf {z = wx + b}$可分解为$\\textbf{y = wx}$和$\\textbf{z = y + b}$，其计算图如图3-3所示，图中`MUL`，`ADD`都是算子，$\\textbf{w}$，$\\textbf{x}$，$\\textbf{b}$即变量。\n",
    "\n",
    "![图3-3:computation graph](imgs/com_graph.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如上有向无环图中，$\\textbf{X}$和$\\textbf{b}$是叶子节点（leaf node），这些节点通常由用户自己创建，不依赖于其他变量。$\\textbf{z}$称为根节点，是计算图的最终目标。利用链式法则很容易求得各个叶子节点的梯度。\n",
    "$${\\partial z \\over \\partial b} = 1,\\space {\\partial z \\over \\partial y} = 1\\\\\n",
    "{\\partial y \\over \\partial w }= x,{\\partial y \\over \\partial x}= w\\\\\n",
    "{\\partial z \\over \\partial x}= {\\partial z \\over \\partial y} {\\partial y \\over \\partial x}=1 * w\\\\\n",
    "{\\partial z \\over \\partial w}= {\\partial z \\over \\partial y} {\\partial y \\over \\partial w}=1 * x\\\\\n",
    "$$\n",
    "而有了计算图，上述链式求导即可利用计算图的反向传播自动完成，其过程如图3-4所示。\n",
    "\n",
    "![图3-4：计算图的反向传播](imgs/com_graph_backward.svg)\n",
    "\n",
    "\n",
    "在PyTorch实现中，autograd会随着用户的操作，记录生成当前variable的所有操作，并由此建立一个有向无环图。用户每进行一个操作，相应的计算图就会发生改变。更底层的实现中，图中记录了操作`Function`，每一个变量在图中的位置可通过其`grad_fn`属性在图中的位置推测得到。在反向传播过程中，autograd沿着这个图从当前变量（根节点$\\textbf{z}$）溯源，可以利用链式求导法则计算所有叶子节点的梯度。每一个前向传播操作的函数都有与之对应的反向传播函数用来计算输入的各个variable的梯度，这些函数的函数名通常以`Backward`结尾。下面结合代码学习autograd的实现细节。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:24:59.427013Z",
     "start_time": "2020-12-09T03:24:59.422025Z"
    }
   },
   "outputs": [],
   "source": [
    "x = t.ones(1)\n",
    "b = t.rand(1, requires_grad = True)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "y = w * x # 等价于y=w.mul(x)\n",
    "z = y + b # 等价于z=y.add(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:25:06.875525Z",
     "start_time": "2020-12-09T03:25:06.871534Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True, True)"
      ]
     },
     "execution_count": 157,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.requires_grad, b.requires_grad, w.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:25:18.079512Z",
     "start_time": "2020-12-09T03:25:18.074524Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 158,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 虽然未指定y.requires_grad为True，但由于y依赖于需要求导的w\n",
    "# 故而y.requires_grad为True\n",
    "y.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:25:24.812811Z",
     "start_time": "2020-12-09T03:25:24.807854Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.is_leaf, w.is_leaf, b.is_leaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:25:31.533856Z",
     "start_time": "2020-12-09T03:25:31.529834Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, False)"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.is_leaf, z.is_leaf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:25:38.621064Z",
     "start_time": "2020-12-09T03:25:38.617074Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AddBackward0 at 0x1fd051bd748>"
      ]
     },
     "execution_count": 161,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# grad_fn可以查看这个variable的反向传播函数，\n",
    "# z是add函数的输出，所以它的反向传播函数是AddBackward\n",
    "z.grad_fn "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:25:54.062733Z",
     "start_time": "2020-12-09T03:25:54.055752Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 162,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# variable的grad_fn对应着和图中的function相对应\n",
    "z.grad_fn.next_functions[0][0] == y.grad_fn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:26:12.940431Z",
     "start_time": "2020-12-09T03:26:12.936441Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((<AccumulateGrad at 0x1fd051bdeb8>, 0), (None, 0))"
      ]
     },
     "execution_count": 163,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 第一个是w，叶子节点，需要求导，梯度是累加的\n",
    "# 第二个是x，叶子节点，不需要求导，所以为None\n",
    "y.grad_fn.next_functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:26:36.667484Z",
     "start_time": "2020-12-09T03:26:36.663462Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, None)"
      ]
     },
     "execution_count": 164,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 叶子节点的grad_fn是None\n",
    "w.grad_fn,x.grad_fn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算w的梯度的时候，需要用到x的数值(${\\partial y\\over \\partial w} = x $)，这些数值在前向过程中会保存成buffer，在计算完梯度之后会自动清空。为了能够多次反向传播需要指定`retain_graph`来保留这些buffer。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:27:05.265589Z",
     "start_time": "2020-12-09T03:27:05.260603Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.])"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 使用retain_graph来保存buffer\n",
    "z.backward(retain_graph=True)\n",
    "w.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 166,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:27:34.413750Z",
     "start_time": "2020-12-09T03:27:34.408794Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2.])"
      ]
     },
     "execution_count": 166,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 多次反向传播，梯度累加，这也就是w中AccumulateGrad标识的含义\n",
    "z.backward()\n",
    "w.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PyTorch使用的是动态图，它的计算图在每次前向传播时都是从头开始构建，所以它能够使用Python控制语句（如for、if等）根据需求创建计算图。这点在自然语言处理领域中很有用，它意味着你不需要事先构建所有可能用到的图的路径，图在运行时才构建。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:28:50.656623Z",
     "start_time": "2020-12-09T03:28:50.651609Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.])"
      ]
     },
     "execution_count": 167,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def abs(x):\n",
    "    if x.data[0]>0: return x\n",
    "    else: return -x\n",
    "x = t.ones(1,requires_grad=True)\n",
    "y = abs(x)\n",
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:29:11.470980Z",
     "start_time": "2020-12-09T03:29:11.462493Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([-1.])\n"
     ]
    }
   ],
   "source": [
    "x = -1*t.ones(1)\n",
    "x = x.requires_grad_()\n",
    "y = abs(x)\n",
    "y.backward()\n",
    "print(x.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:29:18.381307Z",
     "start_time": "2020-12-09T03:29:18.376321Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([1.], grad_fn=<NegBackward>)"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:29:26.655458Z",
     "start_time": "2020-12-09T03:29:26.650501Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([-1.], requires_grad=True)"
      ]
     },
     "execution_count": 170,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:29:33.611529Z",
     "start_time": "2020-12-09T03:29:33.607511Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:29:46.657262Z",
     "start_time": "2020-12-09T03:29:46.653242Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 172,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x.requires_grad\n",
    "cc=x*3\n",
    "cc.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T03:30:31.503748Z",
     "start_time": "2020-12-09T03:30:31.495768Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([0., 0., 0., 6., 3., 2.])"
      ]
     },
     "execution_count": 173,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def f(x):\n",
    "    result = 1\n",
    "    for ii in x:\n",
    "        if ii.item()>0: \n",
    "            result=ii*result\n",
    "    return result\n",
    "x = t.arange(-2,4,dtype=t.float32).requires_grad_()\n",
    "y = f(x) # y = x[3]*x[4]*x[5]\n",
    "y.backward()\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "变量的`requires_grad`属性默认为False，如果某一个节点requires_grad被设置为True，那么所有依赖它的节点`requires_grad`都是True。这其实很好理解，对于$ \\textbf{x}\\to \\textbf{y} \\to \\textbf{z}$，x.requires_grad = True，当需要计算$\\partial z \\over \\partial x$时，根据链式法则，$\\frac{\\partial z}{\\partial x} = \\frac{\\partial z}{\\partial y} \\frac{\\partial y}{\\partial x}$，自然也需要求$ \\frac{\\partial z}{\\partial y}$，所以y.requires_grad会被自动标为True. \n",
    "\n",
    "\n",
    "\n",
    "有些时候我们可能不希望autograd对tensor求导。认为求导需要缓存许多中间结构，增加额外的内存/显存开销，那么我们可以关闭自动求导。对于不需要反向传播的情景（如inference，即测试推理时），关闭自动求导可实现一定程度的速度提升，并节省约一半显存，因其不需要分配空间计算梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:15:26.000915Z",
     "start_time": "2020-12-09T04:15:25.994963Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "execution_count": 174,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.ones(1, requires_grad=True)\n",
    "w = t.rand(1, requires_grad=True)\n",
    "y = x * w\n",
    "# y依赖于w，而w.requires_grad = True\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:15:46.013324Z",
     "start_time": "2020-12-09T04:15:46.007340Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(False, True, False)"
      ]
     },
     "execution_count": 175,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with t.no_grad():\n",
    "    x = t.ones(1)\n",
    "    w = t.rand(1, requires_grad = True)\n",
    "    y = x * w\n",
    "# y依赖于w和x，虽然w.requires_grad = True，但是y的requires_grad依旧为False\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:15:53.246108Z",
     "start_time": "2020-12-09T04:15:53.174297Z"
    }
   },
   "outputs": [],
   "source": [
    "t.no_grad??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:17:09.100427Z",
     "start_time": "2020-12-09T04:17:09.095441Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch.autograd.grad_mode.set_grad_enabled at 0x1fd04d07320>"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 恢复默认配置\n",
    "t.set_grad_enabled(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们想要修改tensor的数值，但是又不希望被autograd记录，那么我么可以对tensor.data进行操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:17:38.762689Z",
     "start_time": "2020-12-09T04:17:38.757702Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.],\n",
       "        [1., 1., 1., 1.]])"
      ]
     },
     "execution_count": 178,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a = t.ones(3,4,requires_grad=True)\n",
    "b = t.ones(3,4,requires_grad=True)\n",
    "c = a * b\n",
    "\n",
    "a.data # 还是一个tensor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:17:48.442746Z",
     "start_time": "2020-12-09T04:17:48.438756Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.data.requires_grad # 但是已经是独立于计算图之外"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:18:02.363776Z",
     "start_time": "2020-12-09T04:18:02.358801Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 180,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = a.data.sigmoid_() # sigmoid_ 是个inplace操作，会修改a自身的值\n",
    "d.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:18:09.534890Z",
     "start_time": "2020-12-09T04:18:09.529875Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.7310585976, 0.7310585976, 0.7310585976, 0.7310585976],\n",
       "        [0.7310585976, 0.7310585976, 0.7310585976, 0.7310585976],\n",
       "        [0.7310585976, 0.7310585976, 0.7310585976, 0.7310585976]],\n",
       "       requires_grad=True)"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果我们希望对tensor，但是又不希望被记录, 可以使用tensor.data 或者tensor.detach()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:18:30.589373Z",
     "start_time": "2020-12-09T04:18:30.585352Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 182,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:18:44.426941Z",
     "start_time": "2020-12-09T04:18:44.422952Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 近似于 tensor=a.data, 但是如果tensor被修改，backward可能会报错\n",
    "tensor = a.detach()\n",
    "tensor.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:18:54.051896Z",
     "start_time": "2020-12-09T04:18:54.043953Z"
    }
   },
   "outputs": [],
   "source": [
    "# 统计tensor的一些指标，不希望被记录\n",
    "mean = tensor.mean()\n",
    "std = tensor.std()\n",
    "maximum = tensor.max()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:19:03.802038Z",
     "start_time": "2020-12-09T04:19:03.799054Z"
    }
   },
   "outputs": [],
   "source": [
    "tensor[0]=1\n",
    "# 下面会报错：　RuntimeError: one of the variables needed for gradient\n",
    "#             computation has been modified by an inplace operation\n",
    "#　因为 c=a*b, b的梯度取决于a，现在修改了tensor，其实也就是修改了a，梯度不再准确\n",
    "# c.sum().backward() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在反向传播过程中非叶子节点的导数计算完之后即被清空。若想查看这些变量的梯度，有两种方法：\n",
    "- 使用autograd.grad函数\n",
    "- 使用hook\n",
    "\n",
    "`autograd.grad`和`hook`方法都是很强大的工具，更详细的用法参考官方api文档，这里举例说明基础的使用。推荐使用`hook`方法，但是在实际使用中应尽量避免修改grad的值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:20:13.581016Z",
     "start_time": "2020-12-09T04:20:13.575032Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(True, True, True)"
      ]
     },
     "execution_count": 187,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.ones(3, requires_grad=True)\n",
    "w = t.rand(3, requires_grad=True)\n",
    "y = x * w\n",
    "# y依赖于w，而w.requires_grad = True\n",
    "z = y.sum()\n",
    "x.requires_grad, w.requires_grad, y.requires_grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:20:26.860629Z",
     "start_time": "2020-12-09T04:20:26.854668Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([0.8198856115, 0.2030361295, 0.8434767127]),\n",
       " tensor([1., 1., 1.]),\n",
       " None)"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 非叶子节点grad计算完之后自动清空，y.grad是None\n",
    "z.backward()\n",
    "(x.grad, w.grad, y.grad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:20:47.019540Z",
     "start_time": "2020-12-09T04:20:47.014522Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([1., 1., 1.]),)"
      ]
     },
     "execution_count": 189,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 第一种方法：使用grad获取中间变量的梯度\n",
    "x = t.ones(3, requires_grad=True)\n",
    "w = t.rand(3, requires_grad=True)\n",
    "y = x * w\n",
    "z = y.sum()\n",
    "# z对y的梯度，隐式调用backward()\n",
    "t.autograd.grad(z, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:21:24.194007Z",
     "start_time": "2020-12-09T04:21:24.183006Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y的梯度： tensor([1., 1., 1.])\n"
     ]
    }
   ],
   "source": [
    "# 第二种方法：使用hook\n",
    "# hook是一个函数，输入是梯度，不应该有返回值\n",
    "def variable_hook(grad):\n",
    "    print('y的梯度：',grad)\n",
    "\n",
    "x = t.ones(3, requires_grad=True)\n",
    "w = t.rand(3, requires_grad=True)\n",
    "y = x * w\n",
    "# 注册hook\n",
    "hook_handle = y.register_hook(variable_hook)\n",
    "z = y.sum()\n",
    "z.backward()\n",
    "\n",
    "# 除非你每次都要用hook，否则用完之后记得移除hook\n",
    "hook_handle.remove()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后再来看看variable中grad属性和backward函数`grad_variables`参数的含义，这里直接下结论：\n",
    "\n",
    "- variable $\\textbf{x}$的梯度是目标函数${f(x)} $对$\\textbf{x}$的梯度，$\\frac{df(x)}{dx} = (\\frac {df(x)}{dx_0},\\frac {df(x)}{dx_1},...,\\frac {df(x)}{dx_N})$，形状和$\\textbf{x}$一致。\n",
    "- 对于y.backward(grad_variables)中的grad_variables相当于链式求导法则中的$\\frac{\\partial z}{\\partial x} = \\frac{\\partial z}{\\partial y} \\frac{\\partial y}{\\partial x}$中的$\\frac{\\partial z}{\\partial y}$。z是目标函数，一般是一个标量，故而$\\frac{\\partial z}{\\partial y}$的形状与variable $\\textbf{y}$的形状一致。`z.backward()`在一定程度上等价于y.backward(grad_y)。`z.backward()`省略了grad_variables参数，是因为$z$是一个标量，而$\\frac{\\partial z}{\\partial z} = 1$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:23:03.982887Z",
     "start_time": "2020-12-09T04:23:03.975934Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2., 4., 6.])"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.arange(0,3, requires_grad=True,dtype=t.float)\n",
    "y = x**2 + x*2\n",
    "z = y.sum()\n",
    "z.backward() # 从z开始反向传播\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T04:23:38.940612Z",
     "start_time": "2020-12-09T04:23:38.934628Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([2., 4., 6.])"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.arange(0,3, requires_grad=True,dtype=t.float)\n",
    "y = x**2 + x*2\n",
    "z = y.sum()\n",
    "y_gradient = t.Tensor([1,1,1]) # dz/dy\n",
    "y.backward(y_gradient) #从y开始反向传播\n",
    "x.grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外值得注意的是，只有对variable的操作才能使用autograd，如果对variable的data直接进行操作，将无法使用反向传播。除了对参数初始化，一般我们不会修改variable.data的值。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在PyTorch中计算图的特点可总结如下：\n",
    "\n",
    "- autograd根据用户对variable的操作构建其计算图。对变量的操作抽象为`Function`。\n",
    "- 对于那些不是任何函数(Function)的输出，由用户创建的节点称为叶子节点，叶子节点的`grad_fn`为None。叶子节点中需要求导的variable，具有`AccumulateGrad`标识，因其梯度是累加的。\n",
    "- variable默认是不需要求导的，即`requires_grad`属性默认为False，如果某一个节点requires_grad被设置为True，那么所有依赖它的节点`requires_grad`都为True。\n",
    "- variable的`volatile`属性默认为False，如果某一个variable的`volatile`属性被设为True，那么所有依赖它的节点`volatile`属性都为True。volatile属性为True的节点不会求导，volatile的优先级比`requires_grad`高。\n",
    "- 多次反向传播时，梯度是累加的。反向传播的中间缓存会被清空，为进行多次反向传播需指定`retain_graph`=True来保存这些缓存。\n",
    "- 非叶子节点的梯度计算完之后即被清空，可以使用`autograd.grad`或`hook`技术获取非叶子节点的值。\n",
    "- variable的grad与data形状一致，应避免直接修改variable.data，因为对data的直接操作无法利用autograd进行反向传播\n",
    "- 反向传播函数`backward`的参数`grad_variables`可以看成链式求导的中间结果，如果是标量，可以省略，默认为1\n",
    "- PyTorch采用动态图设计，可以很方便地查看中间层的输出，动态的设计计算图结构。\n",
    "\n",
    "这些知识不懂大多数情况下也不会影响对pytorch的使用，但是掌握这些知识有助于更好的理解pytorch，并有效的避开很多陷阱"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2.3 扩展autograd\n",
    "\n",
    "\n",
    "目前绝大多数函数都可以使用`autograd`实现反向求导，但如果需要自己写一个复杂的函数，不支持自动反向求导怎么办? 写一个`Function`，实现它的前向传播和反向传播代码，`Function`对应于计算图中的矩形， 它接收参数，计算并返回结果。下面给出一个例子。\n",
    "\n",
    "```python\n",
    "\n",
    "class Mul(Function):\n",
    "                                                            \n",
    "    @staticmethod\n",
    "    def forward(ctx, w, x, b, x_requires_grad = True):\n",
    "        ctx.x_requires_grad = x_requires_grad\n",
    "        ctx.save_for_backward(w,x)\n",
    "        output = w * x + b\n",
    "        return output\n",
    "        \n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output):\n",
    "        w,x = ctx.saved_tensors\n",
    "        grad_w = grad_output * x\n",
    "        if ctx.x_requires_grad:\n",
    "            grad_x = grad_output * w\n",
    "        else:\n",
    "            grad_x = None\n",
    "        grad_b = grad_output * 1\n",
    "        return grad_w, grad_x, grad_b, None\n",
    "```\n",
    "\n",
    "分析如下：\n",
    "\n",
    "- 自定义的Function需要继承autograd.Function，没有构造函数`__init__`，forward和backward函数都是静态方法\n",
    "- backward函数的输出和forward函数的输入一一对应，backward函数的输入和forward函数的输出一一对应\n",
    "- backward函数的grad_output参数即t.autograd.backward中的`grad_variables`\n",
    "- 如果某一个输入不需要求导，直接返回None，如forward中的输入参数x_requires_grad显然无法对它求导，直接返回None即可\n",
    "- 反向传播可能需要利用前向传播的某些中间结果，需要进行保存，否则前向传播结束后这些对象即被释放\n",
    "\n",
    "Function的使用利用Function.apply(variable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:40:58.354602Z",
     "start_time": "2020-12-09T05:40:58.348588Z"
    }
   },
   "outputs": [],
   "source": [
    "from torch.autograd import Function\n",
    "class MultiplyAdd(Function):\n",
    "                                                            \n",
    "    @staticmethod\n",
    "    def forward(ctx, w, x, b):                              \n",
    "        ctx.save_for_backward(w,x)\n",
    "        output = w * x + b\n",
    "        return output\n",
    "        \n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output):                         \n",
    "        w,x = ctx.saved_tensors\n",
    "        grad_w = grad_output * x\n",
    "        grad_x = grad_output * w\n",
    "        grad_b = grad_output * 1\n",
    "        return grad_w, grad_x, grad_b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:41:12.708350Z",
     "start_time": "2020-12-09T05:41:12.695350Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(None, tensor([1.]), tensor([1.]))"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.ones(1)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "b = t.rand(1, requires_grad = True)\n",
    "# 开始前向传播\n",
    "z=MultiplyAdd.apply(w, x, b)\n",
    "# 开始反向传播\n",
    "z.backward()\n",
    "\n",
    "# x不需要求导，中间过程还是会计算它的导数，但随后被清空\n",
    "x.grad, w.grad, b.grad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:41:28.162502Z",
     "start_time": "2020-12-09T05:41:28.152532Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([1.]), tensor([0.7080577612], grad_fn=<MulBackward0>), tensor([1.]))"
      ]
     },
     "execution_count": 198,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.ones(1)\n",
    "w = t.rand(1, requires_grad = True)\n",
    "b = t.rand(1, requires_grad = True)\n",
    "#print('开始前向传播')\n",
    "z=MultiplyAdd.apply(w,x,b)\n",
    "#print('开始反向传播')\n",
    "\n",
    "# 调用MultiplyAdd.backward\n",
    "# 输出grad_w, grad_x, grad_b\n",
    "z.grad_fn.apply(t.ones(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "之所以forward函数的输入是tensor，而backward函数的输入是variable，是为了实现高阶求导。backward函数的输入输出虽然是variable，但在实际使用时autograd.Function会将输入variable提取为tensor，并将计算结果的tensor封装成variable返回。在backward函数中，之所以也要对variable进行操作，是为了能够计算梯度的梯度（backward of backward）。下面举例说明，有关torch.autograd.grad的更详细使用请参照文档。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:42:59.931089Z",
     "start_time": "2020-12-09T05:42:59.925135Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([10.], grad_fn=<MulBackward0>),)"
      ]
     },
     "execution_count": 199,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x = t.tensor([5], requires_grad=True,dtype=t.float)\n",
    "y = x ** 2\n",
    "grad_x = t.autograd.grad(y, x, create_graph=True)\n",
    "grad_x # dy/dx = 2 * x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:43:14.860146Z",
     "start_time": "2020-12-09T05:43:14.852196Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([2.]),)"
      ]
     },
     "execution_count": 200,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grad_grad_x = t.autograd.grad(grad_x[0],x)\n",
    "grad_grad_x # 二阶导数 d(2x)/dx = 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这种设计虽然能让`autograd`具有高阶求导功能，但其也限制了Tensor的使用，因autograd中反向传播的函数只能利用当前已经有的Variable操作。这个设计是在`0.2`版本新加入的，为了更好的灵活性，也为了兼容旧版本的代码，PyTorch还提供了另外一种扩展autograd的方法。PyTorch提供了一个装饰器`@once_differentiable`，能够在backward函数中自动将输入的variable提取成tensor，把计算结果的tensor自动封装成variable。有了这个特性我们就能够很方便的使用numpy/scipy中的函数，操作不再局限于variable所支持的操作。但是这种做法正如名字中所暗示的那样只能求导一次，它打断了反向传播图，不再支持高阶求导。\n",
    "\n",
    "\n",
    "上面所描述的都是新式Function，还有个legacy Function，可以带有`__init__`方法，`forward`和`backwad`函数也不需要声明为`@staticmethod`，但随着版本更迭，此类Function将越来越少遇到，在此不做更多介绍。\n",
    "\n",
    "此外在实现了自己的Function之后，还可以使用`gradcheck`函数来检测实现是否正确。`gradcheck`通过数值逼近来计算梯度，可能具有一定的误差，通过控制`eps`的大小可以控制容忍的误差。\n",
    "关于这部份的内容可以参考github上开发者们的讨论[^3]。\n",
    "\n",
    "[^3]: https://github.com/pytorch/pytorch/pull/1016"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下面举例说明如何利用Function实现sigmoid Function。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:45:10.222237Z",
     "start_time": "2020-12-09T05:45:10.216253Z"
    }
   },
   "outputs": [],
   "source": [
    "class Sigmoid(Function):\n",
    "                                                             \n",
    "    @staticmethod\n",
    "    def forward(ctx, x): \n",
    "        output = 1 / (1 + t.exp(-x))\n",
    "        ctx.save_for_backward(output)\n",
    "        return output\n",
    "        \n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output): \n",
    "        output,  = ctx.saved_tensors\n",
    "        grad_x = output * (1 - output) * grad_output\n",
    "        return grad_x                            "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:46:54.066495Z",
     "start_time": "2020-12-09T05:46:54.040595Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 202,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 采用数值逼近方式检验计算梯度的公式对不对\n",
    "test_input = t.randn(3,4, requires_grad=True).double()\n",
    "t.autograd.gradcheck(Sigmoid.apply, (test_input,), eps=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:47:28.357983Z",
     "start_time": "2020-12-09T05:47:27.845729Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "258 µs ± 25.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n",
      "244 µs ± 28.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n",
      "203 µs ± 17.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
     ]
    }
   ],
   "source": [
    "def f_sigmoid(x):\n",
    "    y = Sigmoid.apply(x)\n",
    "    y.backward(t.ones(x.size()))\n",
    "    \n",
    "def f_naive(x):\n",
    "    y =  1/(1 + t.exp(-x))\n",
    "    y.backward(t.ones(x.size()))\n",
    "    \n",
    "def f_th(x):\n",
    "    y = t.sigmoid(x)\n",
    "    y.backward(t.ones(x.size()))\n",
    "    \n",
    "x=t.randn(100, 100, requires_grad=True)\n",
    "%timeit -n 100 f_sigmoid(x)\n",
    "%timeit -n 100 f_naive(x)\n",
    "%timeit -n 100 f_th(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "显然`f_sigmoid`要比单纯利用`autograd`加减和乘方操作实现的函数快不少，因为f_sigmoid的backward优化了反向传播的过程。另外可以看出系统实现的built-in接口(t.sigmoid)更快。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2.4 小试牛刀: 用Variable实现线性回归\n",
    "在上一节中讲解了利用tensor实现线性回归，在这一小节中，将讲解如何利用autograd/Variable实现线性回归，以此感受autograd的便捷之处。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:48:39.920185Z",
     "start_time": "2020-12-09T05:48:39.908217Z"
    }
   },
   "outputs": [],
   "source": [
    "import torch as t\n",
    "%matplotlib inline\n",
    "from matplotlib import pyplot as plt\n",
    "from IPython import display \n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:48:45.787942Z",
     "start_time": "2020-12-09T05:48:45.780941Z"
    }
   },
   "outputs": [],
   "source": [
    "# 设置随机数种子，为了在不同人电脑上运行时下面的输出一致\n",
    "t.manual_seed(1000) \n",
    "\n",
    "def get_fake_data(batch_size=8):\n",
    "    ''' 产生随机数据：y = x*2 + 3，加上了一些噪声'''\n",
    "    x = t.rand(batch_size,1) * 5\n",
    "    y = x * 2 + 3 + t.randn(batch_size, 1)\n",
    "    return x, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 206,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:48:52.162895Z",
     "start_time": "2020-12-09T05:48:51.924561Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1fd05014668>"
      ]
     },
     "execution_count": 206,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPrUlEQVR4nO3dX2xk5X3G8eep1yheQmvCDpQ1bDeRkNWGFHZrIf60KCmlBkLCBuUC1FRpFNVqlbbQC1dsL4J6lVbuRdqLtlpRWqImRCnxbiuUYFBSmqoUIi+G7NKNC6FAsGnWlDiEMBK77q8XMyazE9vz5xzPOe/M9yONPH7P8Zyf3n39+Jz3/FlHhAAA6fmpogsAAHSHAAeARBHgAJAoAhwAEkWAA0CidvRyY7t27Yq9e/f2cpMAkLyjR4++GhGV5vaeBvjevXs1Pz/fy00CQPJsv7hRe8spFNv32j5p+3hD27tsP2L72frXc/MsFgDQWjtz4H8v6YamtrskfS0iLpH0tfr3AIAeahngEfENSa81Nd8i6b76+/skHci3LABAK91ehXJBRLwiSfWv52+2ou0p2/O251dWVrrcHACg2bZfRhgRhyJiIiImKpWfOIkKAOhSt1ehfM/2hRHxiu0LJZ3MsygA6BdHFpY0M7eo5dWqdo+OaHpyXAf2jeXy2d3ugf+zpI/X339c0j/lUg0A9JEjC0s6OHtMS6tVhaSl1aoOzh7TkYWlXD6/ncsI75f0H5LGbb9s+5OS/lTS9baflXR9/XsAQIOZuUVVT62d0VY9taaZucVcPr/lFEpE3L7JoutyqQAA+tTyarWj9k7xLBQA2Ca7R0c6au8UAQ4A22R6clwjw0NntI0MD2l6cjyXz+/ps1AAYJCsX22yXVehEOAAsI0O7BvLLbCbMYUCAIkiwAEgUQQ4ACSKAAeARBHgAJAoAhwAEkWAA0CiCHAASBQBDgCJIsABIFEEOAAkigAHgEQR4ACQKAIcABJFgANAoghwAEgUAQ4AiSLAASBRBDgAJCpTgNu+w/Zx28/YvjOnmgAAbeg6wG1fKum3JV0h6TJJN9u+JK/CAABby7IH/vOSHo+INyPitKR/lfSRfMoCALSSJcCPS7rW9nm2d0q6SdLF+ZQFAGhlR7c/GBEnbP+ZpEckvSHpaUmnm9ezPSVpSpL27NnT7eYAAE0yncSMiL+NiP0Rca2k1yQ9u8E6hyJiIiImKpVKls0BABp0vQcuSbbPj4iTtvdIulXSVfmUBQBoJVOAS/qy7fMknZL0qYj4fg41AQDakCnAI+JX8ioEANAZ7sQEgEQR4ACQKAIcABJFgANAoghwAEgUAQ4AiSLAASBRBDgAJIoAB4BEZb2VHkjWkYUlzcwtanm1qt2jI5qeHNeBfWNFlwW0jQDHQDqysKSDs8dUPbUmSVparerg7DFJIsSRDKZQMJBm5hbfDu911VNrmplbLKgioHMEOAbS8mq1o3agjAhwDKTdoyMdtQNlRIBjIE1PjmtkeOiMtpHhIU1PjhdUEdA5TmJiIK2fqOQqFKSMAMfAOrBvjMBG0phCAYBEEeAAkCgCHAASRYADQKIIcABIFAEOAIkiwAEgUZkC3PYf2n7G9nHb99t+R16FAQC21nWA2x6T9AeSJiLiUklDkm7LqzAAwNayTqHskDRie4eknZKWs5cEAGhH1wEeEUuS/lzSS5JekfSDiHi4eT3bU7bnbc+vrKx0XykA4AxZplDOlXSLpHdL2i3pbNsfa14vIg5FxERETFQqle4rBQCcIcsUyq9J+u+IWImIU5JmJV2dT1kAgFayBPhLkq60vdO2JV0n6UQ+ZQEAWskyB/6EpAckPSnpWP2zDuVUFwCghUzPA4+IuyXdnVMtAIAOcCcmACSKAAeARBHgAJAoAhwAEkWAA0CiCHAASBQBDgCJIsABIFGZbuRBfziysKSZuUUtr1a1e3RE05PjOrBvrOiyALRAgA+4IwtLOjh7TNVTa5KkpdWqDs4ekyRCHCg5AnzAzcwtvh3e66qn1jQzt0iAA5soy1ErAT7gllerHbUDg65MR62cxBxwu0dHOmoHBt1WR629RoAPuOnJcY0MD53RNjI8pOnJ8YIqAsqtTEetBPiAO7BvTJ+59X0aGx2RJY2Njugzt76P+W9gE2U6amUOHDqwb4zABto0PTl+xhy4VNxRKwEOAB1Y39nhKhQASFBZjlqZAweARLEH3kNlufgfQH8gwHukTBf/A+gPTKH0SJku/gfQHwjwHinTxf8A+kPXAW573PZTDa/Xbd+ZY219pUwX/wPoD10HeEQsRsTlEXG5pF+S9Kakw3kV1m+4ZR1A3vI6iXmdpO9ExIs5fV7fKdPF/wD6Q14Bfpuk+zdaYHtK0pQk7dmzJ6fNpaksF/8D6A+ZT2LaPkvShyX940bLI+JQRExExESlUsm6OQBAXR5Xodwo6cmI+F4OnwUAaFMeAX67Npk+AQBsn0wBbnunpOslzeZTDgCgXZlOYkbEm5LOy6kWAEAHuBMTABJFgANAoghwAEgUj5PFQOMZ7UgZAY6BxTPakTqmUDCweEY7UkeAY2DxjHakjgDHwOIZ7UgdAY6BxTPakTpOYmJg8Yx2pI4Ax0DjGe1IGVMoAJCo0u+Bc6MFAGys1AHOjRYAsLlST6FwowUAbK7UAc6NFgCwuVIHODdaAMDmSh3g3GgBAJsr9UlMbrQAgM2VOsAlbrQAgM2UegoFALA5AhwAEkWAA0CiMgW47VHbD9j+tu0Ttq/KqzAAwNaynsT8C0kPRcRHbZ8laWcONQEA2tB1gNv+aUnXSvotSYqItyS9lU9ZAIBWskyhvEfSiqS/s71g+x7bZzevZHvK9rzt+ZWVlQybAwA0yhLgOyTtl/TXEbFP0o8k3dW8UkQcioiJiJioVCoZNgcAaJQlwF+W9HJEPFH//gHVAh0A0ANdB3hE/I+k79pefzDJdZL+M5eqAAAtZb0K5fclfb5+Bcrzkj6RvSQAQDsyBXhEPCVpIp9SAACd4E5MAEgUAQ4AiSLAASBRBDgAJIoAB4BEEeAAkCgCHAASRYADQKIIcABIFAEOAIkiwAEgUQQ4ACSKAAeARBHgAJAoAhwAEkWAA0CiCHAASBQBDgCJIsABIFEEOAAkigAHgEQR4ACQKAIcABJFgANAonZk+WHbL0j6oaQ1SacjYiKPogAArWUK8LoPRMSrOXwOAKADTKEAQKKyBnhIetj2UdtTG61ge8r2vO35lZWVjJsDAKzLGuDXRMR+STdK+pTta5tXiIhDETEREROVSiXj5gAA6zIFeEQs17+elHRY0hV5FAUAaK3rALd9tu1z1t9L+nVJx/MqDACwtSxXoVwg6bDt9c/5QkQ8lEtVAICWug7wiHhe0mU51gIA6ACXEQJAoghwAEgUAQ4AiSLAASBRBDgAJIoAB4BEEeAAkCgCHAASRYADQKIIcABIFAEOAIkiwAEgUQQ4ACSKAAeARBHgAJAoAhwAEkWAA0CiCHAASBQBDgCJIsABIFEEOAAkigAHgEQR4ACQqMwBbnvI9oLtB/MoCADQnjz2wO+QdCKHzwEAdCBTgNu+SNIHJd2TTzkAgHZl3QP/rKQ/kvR/m61ge8r2vO35lZWVjJsDAKzrOsBt3yzpZEQc3Wq9iDgUERMRMVGpVLrdHACgSZY98Gskfdj2C5K+KOlXbf9DLlUBAFrqOsAj4mBEXBQReyXdJunrEfGx3CoDAGyJ68ABIFE78viQiHhU0qN5fBYAoD25BHiZHFlY0szcopZXq9o9OqLpyXEd2DdWdFkAkLu+CvAjC0s6OHtM1VNrkqSl1aoOzh6TJEIcQN/pqznwmbnFt8N7XfXUmmbmFguqCAC2T18F+PJqtaN2AEhZXwX47tGRjtoBIGV9FeDTk+MaGR46o21keEjTk+MFVQQA26evTmKun6jkKhQAg6CvAlyqhTiBDWAQ9NUUCgAMEgIcABJFgANAoghwAEgUAQ4AiXJE9G5j9oqkF1ustkvSqz0oJwtqzE8KdVJjflKos4w1/lxE/MR/adbTAG+H7fmImCi6jq1QY35SqJMa85NCnSnUuI4pFABIFAEOAIkqY4AfKrqANlBjflKokxrzk0KdKdQoqYRz4ACA9pRxDxwA0AYCHAAS1ZMAt32v7ZO2j2+y/Ddsf6v+esz2ZQ3LXrB9zPZTtucLrvP9tn9Qr+Up259uWHaD7UXbz9m+q8AapxvqO257zfa76st60pe2L7b9L7ZP2H7G9h0brGPbf1nvr2/Z3t+wbNv7ss0aCx+XbdZZ6Lhss8ZCx6Xtd9j+pu2n6zX+yQbrFDomuxIR2/6SdK2k/ZKOb7L8aknn1t/fKOmJhmUvSNpVkjrfL+nBDdqHJH1H0nsknSXpaUm/UESNTet+SNLXe92Xki6UtL/+/hxJ/9XcH5JukvRVSZZ05fq/ea/6ss0aCx+XbdZZ6Lhsp8aix2V9nL2z/n5Y0hOSrizTmOzm1ZM98Ij4hqTXtlj+WER8v/7t45Iu6kVdG9SxZZ1buELScxHxfES8JemLkm7Jtbi6Dmu8XdL921HHViLilYh4sv7+h5JOSGp+SPstkj4XNY9LGrV9oXrUl+3UWIZx2WZfbqY0fdmk5+OyPs7eqH87XH81X8FR6JjsRhnnwD+p2l/BdSHpYdtHbU8VVFOjq+qHYV+1/d5625ik7zas87La/yXbFrZ3SrpB0pcbmnvel7b3Stqn2h5Po836rOd9uUWNjQofly3qLMW4bNWXRY5L20O2n5J0UtIjEVHaMdmuUv2PPLY/oNovyi83NF8TEcu2z5f0iO1v1/dCi/Ckas8keMP2TZKOSLpEtUOuZkVfn/khSf8eEY176z3tS9vvVO0X9c6IeL158QY/Elu0b4sWNa6vU/i4bFFnKcZlO32pAsdlRKxJutz2qKTDti+NiMZzSaUYk50ozR647V+UdI+kWyLif9fbI2K5/vWkpMOqHc4UIiJeXz8Mi4ivSBq2vUu1v8gXN6x6kaTlAkpsdJuaDlN72Ze2h1X7Zf58RMxusMpmfdazvmyjxlKMy1Z1lmFcttOXdYWOy/p2ViU9qtqRQKPCx2THejXZLmmvNj85uEfSc5Kubmo/W9I5De8fk3RDgXX+rH5889MVkl5S7a/zDknPS3q3fnyS471F1Fhf/jOqzZOfXURf1vvkc5I+u8U6H9SZJ4y+WW/vSV+2WWPh47LNOgsdl+3UWPS4lFSRNFp/PyLp3yTdXKYx2c2rJ1Motu9X7Uz5LtsvS7pbtZMIioi/kfRpSedJ+ivbknQ6ak8Du0C1Q531TvxCRDxUYJ0flfS7tk9Lqkq6LWr/wqdt/56kOdXOWN8bEc8UVKMkfUTSwxHxo4Yf7WVfXiPpNyUdq885StIfqxaI63V+RbWz/s9JelPSJ+rLetWX7dRYhnHZTp1Fj8t2apSKHZcXSrrP9pBqMw9fiogHbf9OQ41Fj8mOcSs9ACSqNHPgAIDOEOAAkCgCHAASRYADQKIIcABIFAEOAIkiwAEgUf8PdoBTdmOFwaUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 来看看产生x-y分布是什么样的\n",
    "x, y = get_fake_data()\n",
    "plt.scatter(x.squeeze().numpy(), y.squeeze().numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 207,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:49:08.319173Z",
     "start_time": "2020-12-09T05:49:02.228179Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAAAcJ0lEQVR4nO3deXSV9b3v8feXOczKpAQiAhJAQMHYqrQOOICztVZrq7Xalp4OVmulwupdt/ecdc8FxalOVep8tK22Uk9PlUkRUVRqEJVqEgjIFJAwGMKQkOl7/9gBYkhgJ3t49n7257UWy2RnJ883W/isZz379/w+5u6IiEj6axP0ACIiEh8KdBGRkFCgi4iEhAJdRCQkFOgiIiHRLpkH6927tw8aNCiZhxQRiTsHduyporS8kpo6p0dWe/p170THdrGfI5ftraakrIK6BisQa8q2eG1F+RF/eFIDfdCgQeTn5yfzkCIicePuvLric2bOK2T79r1ccvzRTLtoBCcP7Bm3Y4yfsZCasoovPbb5mVstmu9NaqCLiKSr99ZsZ/qcQj7aUMawfl158vt5nJPbF7OosjZqmxqFeUso0EVEDqPo813cObeQhYWlHNujE3ddNYZvjhtA2zbxDfL9+vfMoqSVoa5AFxFpwuadFdw7fyUvfbCRLh3bccek4dw4fhCd2rdNyPFeXl7CzHlFlJRVYESu0x/gXhfNz1Cgi4g0sLOimt8vWs1TSz7DHW4afzw/O2coR3XpkLBjvry8hGmzV1BRXQtEwnx/qGf3zGJD+dZ10fwcBbqICLCvppb/encdD71RzM6Kaq44OZvbzh/GwKM7J/zYM+cVHQjz/faH+ZKpE7Bp5Tui+TkKdBHJaHV1zn9/VMLd81ZSUlbBmcP6cMekXE7s3yNpMzT3RmhL3yBVoItIxlq8cisz5hTy6eZyRmV3566rxjB+aO+kz9HcG6H9e2a16Oco0EUk4/yrZCcz5hTydvE2Bh6dxe++fTKXjulPmwStXDmSKRNzv3QNHSCrfVumTMxt0c9RoItIxtiwYy8z5xXx9482cVTn9vzvS0by3dNy6NguMStXonXF2Gwgci19U1kF/XtmMWVi7oHHo6VAF5HQ27GnigcXruK599bRto3xs3OG8OOzhtC9U/ugRzvgirHZLQ7wxhToIhJaFVW1PLnkMx5dtJo9VTVcnTeQW88bxjE9OgU9WkIo0EUkdGpq6/jrso3c99pKtpTv47wR/bhjUi4n9OsW9GgJdcRAN7MngUuAUncfVf/YTOBSoApYDdzo7mUJnFNE5IjcndcKSrlrbiGrSnczNqcnD31nHKcOOjro0ZIimr0enwYmNXpsATDK3ccAK4FpcZ5LRKRFPlj/BVc/9i4/ejaf2jrn0evGMfsnZ2RMmEMUZ+juvtjMBjV6bH6DT98DrorzXCIiUVm9dTcz5xYx95PP6d21I//3ilFcc+pA2rfNvP6eeFxDvwl4obkvmtlkYDJATk5OHA4nIgKluyr53Wur+PP7G+jUrg23nT+MH3zteLp0zNy3BmP6zc3sN0AN8Hxzz3H3WcAsgLy8PG/ueSIi0di9r4ZZi9fw+FtrqKqp47qv5nDzuSfQu2vHoEcLXKsD3cxuIPJm6bnurqAWkYSqrq3jT/9czwOvr2Lb7iouHnMsUy7IZVDvLkGPljJaFehmNgm4AzjL3ffGdyQRkYMa1r6t3b6X0wYfzRM3jOCkONa+hUU0yxb/BJwN9DazjcBviaxq6QgsqK9fes/d/y2Bc4pIBmpY+5bbrxtPff9Uzs7tE/fat7CIZpXLtU08/EQCZhERAQ6tfZt51RiuTGDtW1hk7tvBIpJyNpVVcO+CSO1b147tmHrhcL5/RuJq38JGgS4igdtZUc0ji4p5esla3OGHX4vUvvXsnLjatzBSoItIYCqrD9a+lVdW842Ts7ntgmEMOCrxtW9hpEAXkaSrq3Ne/rCEe+YfrH2bOmk4I/t3D3q0tKZAF5GkcXcWr9rGjDmFFARc+xZGCnQRSYoVG3cyY24BS4q3p0TtWxgp0EUkoVK19i2MFOgikhCNa99+fs5QJp81OKVq38JGgS4icdW49u2aUyO1b/26h7P2LZUo0EUkLvbXvt27YCWlu/Zx/shI7dvQvsHVvr28vISZ84rYVFZB/55ZTJmYG3MRcypToItITPbXvt05t5Di0t2My+nJw98Nvvbt5eUlTJu9gorqWgBKyiqYNnsFQGhDXYEuIq22bN0XzJhTwPtrv2Bwny48et0pTDyxX0psnjVzXtGBMN+vorqWmfOKFOgiIvut3rqbu+YWMu+TLfTp1pH//MYorskbSLsUqn3bVFbRosfDQIEuIlErLa/k/tdX8UKD2rcffv14OndIvSjp3zOLkibCu3/PrACmSY7U+78gIiln974aZr25mj+89RnVtelR+zZlYu6XrqEDZLVvy5SJuQFOlVgKdBFpVlXNwdq37XvSq/Zt/3VyrXIRkYzm7ryyYjMz5xWxrr727ckL06/27Yqx2aEO8MYU6CLyJe+u3s6MOQV8tHGnat/SjAJdRAAo/LycO+cU8kbRVtW+pSkFukiGa1j71k21b2lNgS6SoXbujdS+PfXOWgB+9PXB/PTsIap9S2MKdJEMU1ldy7PvruXhN1ZHat/GZnPb+ap9CwMFukiGqK1zXl5ewr0LIrVvZw3rwx2qfQsVBbpIyLk7b67cyow5hRR+vovR2T2YedUYzlDtW+gcMdDN7EngEqDU3UfVP3Y08AIwCFgLXO3uXyRuTBFpjRUbdzJ9TgHvrI7Uvj1w7VguGX1sWtW+ZdoWuLGI5gz9aeAh4NkGj00FXnf3GWY2tf7zO+I/noi0xvrte5k5v4j/+WgTR3fpwG8vHcl3v3ocHdoFs3lWa0M5E7fAjcURA93dF5vZoEYPXw6cXf/xM8AiFOgigdu+ex8PLizm+aUHa99+fNZgugVY+xZLKGfiFrixaO019H7uvhnA3TebWd84ziQiLbS3qoYn3/6MR99cw94Uq32LJZQzcQvcWCT8TVEzmwxMBsjJyUn04UQySk1tHX9ZtpH7DlP7FvQ16FhCORO3wI1Fay+obTGzYwHq/1va3BPdfZa757l7Xp8+fVp5OBFpyN2Z98nnTLx/MdNmr2DAUVn89d9O5w/fyzskzKfNXkFJWQXOwcsdLy8vSdqszYVvNKE8ZWIuWY3uWA37FrixaG2g/x24of7jG4D/js84InIky9bt4FuPvsuP/2sZDjx63Sm89JMzyGuiw/NwlzuSJZZQvmJsNtOvHE12zywMyO6ZxfQrR+v6eTOiWbb4JyJvgPY2s43Ab4EZwItm9gNgPfCtRA4pIlBcupuZ81pW+5YK16Bj3Zc807bAjUU0q1yubeZL58Z5FhFpQml5Jfe9tooX81te+5Yq16AVysmhO0VFUtSuympmLV7D4/W1b9efdhw/nzC0RbVvmVjDlskU6CIppqqmjj8uXceDC4vZvqeKS8Ycy5SJuRzXq+W1b5lYw5bJFOgiKcLd+cfHm7l7fqT27fTBvZh64fCYa990uSNzKNBFUsA7q7cxY04hH2/cyfBjuvHUjady9jDVvknLKNBFAlSwuZw75xaySLVvEgcKdJEAlJRVcO/8lcxeHql9m3bhcG5Q7ZvESIEukkSqfZNEUqCLJIFq3yQZFOgiCbS/9u2e+UVs2lmp2jdJKAW6SAK4O4tWbuXOBrVvd3/rJNW+SUIp0EXi7OONZUx/tZB312wn5+jOaVn7JulJgS4SJ+u27+Hu+SsP1L79n0tH8p0Aa99STdD7smcCBbpIjBrXvt08YSiTzwy29i3VqBs0ORToIq20t6qGJ976jMcWr6Giupar8wZy63knpETtW6pRN2hyKNBFWqimto4X8zdy/2uR2rcLRvbj15OGM7Rv16BHS1mpsC97JlCgi0TJ3Zn/6RbumlvI6q17OOW4o3jku+OabAqSL0uVfdnDToEuEoVl63Yw/dVC8td9weA+XXjs+lO4YGQ/bZ4VJe3LnhwKdJHDKC7dzV1zC5n/aaT27f99YzRX5w04bO2bHEr7sieHAl2kCQ1r37Lat+VX5w/jB1HWvknTtC974ulvp2SsptZFnzui74Hat5q6SO3bzROG0qsFtW8iQVGgS0Zqal30lL98RIf2bdizrzam2jeRoCjQJSM1tS66us6xWufvPx/PmAE9gxlMJAYKdElZibxVvLn1z1U1dQpzSVsKdElJibxVvGBzOR3atWFfTd0hX8vWumhJY1p7JSnpcLeKt1ZJWQW/evEjLnrgLdqY0a7R7odaFy3pLqYzdDP7JfBDwIEVwI3uXhmPwSSzxfNW8Z17q3l4UTFP19e+Tf76YH569lDeKCrVumgJlVYHupllA78ARrp7hZm9CHwbeDpOs0kGi8et4pXVtTzzzloefqOYXftquHLsAG67YNiByyqpsi5a28pKvMR6Db0dkGVm1UBnYFPsI4nEdqt4bZ3zt+Ul3Ftf+3Z2bqT2bcSxqVf7pm1lJZ5aHejuXmJmdwPrgQpgvrvPb/w8M5sMTAbIyclp7eEkw7TmVvHGtW9jBvTg7qtP4owhqVv7pm1lJZ5iueRyFHA5cDxQBvzFzK5z9+caPs/dZwGzAPLy8rz1o0qmacklkca1bw9eO5aL06D2TdvKSjzFcsnlPOAzd98KYGazgTOA5w77XSJxtG77HmbOK+IfH29Oy9o3bSsr8RRLoK8HTjOzzkQuuZwL5MdlKpEjaFj71q5Nm7StfdO2shJPsVxDX2pmfwU+AGqA5dRfWhE5ktau7Ghc+3bNqQO59dwT6JumtW/aVlbiydyTd1k7Ly/P8/N1Ep/pGq/sgMhZ6fQrRzcbZI1r3yae2I8pE1X7JpnBzJa5e96Rnqdb/yXpWrKyo3HtW95xR/H768ZxynGqfRNpTIEuSRftyo78tTuYPqeQZeu+YEifLsy6/hTOV+2bSLMU6JJ0R1rZ0bD2rW+3jky/cjTfOkW1byJHokCXpGtuZcfkMwczbfaKA7Vvt18wjJu+lvzaN92KL+lKgZ5iMiFMGq/sOKZHJ0Zl92DGnMLAa990K76kM61ySSGtWf2Rzqpq6vjj0nU8sLCYHXuquPSk/ky5IJecXp0Dm2n8jIVNXg7K7pnFkqkTAphIRKtc0lKm7OtRV+f8Y8Vm7p5XxPodezljSC+mXjg8JZqCdCu+pDMFegrJhDB5p3gb0+cUsqJkJ8OP6cbTN57KWcP6pMzKFd2KL+lMgZ5CwhwmBZvLmTGnkDdXbiW7Zxb3Xn0SV5ycnXKbZ+lWfElnCvQUEsYwKSmr4J75RfxteQndO7XnNxeN4PrTj6NT+7ZBj9Yk3Yov6UyBnkLCFCaH1L6dOZifnjWUHp1Tf/OsVGkyEmkpBXqKSfcwaVz79s1xA/jl+Qdr30QkcRToEheNa9/Oye3DHRcOZ/gxqVf7JhJWCnSJSePat5PSoPZNJKwU6NJqDWvfjuvVmYe+E6l9S5UliCKZRoEuLdaw9q1Xlw78+2Uncu1XctKm9k0krBToErXGtW+/mDCUH6Vh7ZtIWCnQ5YjCVvsmElYKdGmWat9E0osCXQ7h7sz7ZAt3zStkjWrfRNKGAl2+RLVvIulLgS4AFJfu4s65RSxQ7ZtI2lKgZ7gt5ZXc/9pKXnh/A507tAus9k1EYqd/tRlqV2U1j725hsffXkNtnfO90wclvfYtE+r2RJIppkA3s57A48AowIGb3P3dOMwlCVJVU8fzS9fxYMC1b+ruFIm/WM/QfwfMdferzKwDEFwZpBxWqtW+ZUrdnkgytTrQzaw7cCbwfQB3rwKq4jOWxFMq1r5lQt2eSLLFcoY+GNgKPGVmJwHLgFvcfU/DJ5nZZGAyQE5OTgyHk5ZK5dq3MNftiQQlljVp7YBxwO/dfSywB5ja+EnuPsvd89w9r0+fPjEcTqJVUlbBbS9+yEUPvMWHG8r4zUUjeP1XZ3HluAEpEeYQqdvLalRDl+51eyJBi+UMfSOw0d2X1n/+V5oIdEmesr1VPLJodVrUvoWpbk8kVbQ60N39czPbYGa57l4EnAt8Gr/RJFrpWvuW7nV7Iqkm1lUuNwPP169wWQPcGPtIEi3VvolIQzEFurt/COTFZxSJlruzqGgrd849WPt2z9Unc/qQXkGPJiIB0p2iaeajDWVMn1PAe2t2qPZNRL5EgZ4m1m3fw13zinhFtW8i0gwFejNSZZ+Rbbv38eDrq3h+6Xrat1Xtm4g0T4HehFTYZ2RvVQ2Pv/UZj725msqaOtW+icgRKdCbEOQ+IzW1dbyQv4H7X1vFVtW+iUgLKNCbEMQ+I03Vvj2q2jcRaQEFehOSvc+Iat9EJB4U6E2YMjH3S9fQITH7jKj2TUTiSYHehETvM6LaNxFJBHP3pB0sLy/P8/Pzk3a8VNO49u30wb1YuWU3W8ortTmViDTLzJa5+xHvytcpYRI0rn277KT+jBnQg3vmr1QFm4jEjQI9gfbXvs2cV8iGHRWcMaQX0y4cwegBPRg/Y6Eq2EQkrhToCbKkeBszDlP7pgo2EYk3BXqcfbqpnBlzC1l8hNo3VbCJSLwp0ONk4xd7uXf+Sv72YQndO7XnNxeN4PrTj6NTo5q1/ZK1NFJEMocCPUZle6t4+I1innl3HdB07dvhNvpKhQ3A4iFVNjMTyWQK9FaqrK7l6XfW8kiD2rfbzh92yCWTI230FYbQS4XNzEQkJIGezLPD2jpn9gcbuW/Byqhq34Lc6CtZMuF3FEkHaR/oyTo7bG3tWyasZsmE31EkHaR9oCfj7DCW2rdMWM2SCb+jSDpI+12gEnl2uHbbHn72xw+4/OEl/POzHQBU19RRU+tR74Q4ZWIuWY1WuoRtNUsm/I4i6SDtz9ATcXbYsPatjRnt2hg1dZE9bzbtrGzRJZ2wrWZpSib8jiLpIO0352p8DR0iZ4fTrxzd4kBpqvZtYUEpn5dXHvLc7J5ZLJk6Ieb5RUSOJGM254rH2WF1bR0vNqh9m3TiMUyZlMuQPl05fukrTX6P3vATkVST9oEOrV/PHal9+5y75haxZtseTh10FI9edwqnHHfUgefoDT8RSRcxB7qZtQXygRJ3vyT2kZLj/bU7mP5qAR+sL2No36784Xt5nDei7yFvduoWfRFJF/E4Q78FKACavrMmxRSX7mLGnCJeK9hCv+4dmXHlaK46TO2b3vATkXQRU6Cb2QDgYuA/gdviMlGCbCmv5L4FK3kxfwNdOrRjysRcbhp/PFkdmt48q6Gw3KIvIuEW6xn6/cCvgW7NPcHMJgOTAXJycmI8XMuVV1bz2JureeLtz6itc244YxA3TziBo7t0SPosIiKJ1OpAN7NLgFJ3X2ZmZzf3PHefBcyCyLLF1h6vpfbV1PL8e+t5cOEqvthbzWUn9ef2C3LJ6dU5WSOIiCRVLGfo44HLzOwioBPQ3cyec/fr4jNa69TVOf/z8Sbunl/Ehh0VjB/ai6mTIrVvIiJh1upAd/dpwDSA+jP024MO8yXF25g+p4B/lZQz4tjuPHPTaM48oXfUt+mLiKSzUKxDj7b2TUQkzOIS6O6+CFgUj5/VEhu/2Ms981fycn3t2/+6eATXndZ87ZuISJil5Rl62d4qHlpYzLPvrsMMfnzmEH5y9hB6ZLU/8jeLiIRUWgV6ZXUtTy1ZyyOLitm9r4arxg3gl03UvomIZKK0CPTaOuel+tq3zTsrmTC8L3dMGk7uMc0ufxcRyTgpHejuzhtFpdw5p4iiLZHat/uuOZnTBh++9k1EJBOlbKB/uKGM6a8WsPSzHQzq1ZmHvzOOi0YfoyWIIiLNSLlAX7ttDzPnFfHKis306tKB/7j8RK79Sg7tm9k8S0REIlIm0Lft3scDr6/ij0vX06FdG35x7glMPnMwXTumzIgiIikt8LTcsy9S+zZrcaT27dunDuSW806gb7dOQY8mIpJWAgv06to6/vz+Bn732iq27d7HhaOO4faJkdo3ERFpuaQHelO1b49d/+XaNxERabmkBvqefTVc+ft3WL6+jBP6duXx7+VxbhO1byIi0nJJDfQ12/bQvayCO785mm+Oa772TUREWi6pgd6veycW3X5OVLVvIiLSMkk9Re7braPCXEQkQXTNQ0QkJBToIiIhoUAXEQkJBbqISEgo0EVEQkKBLiISEgp0EZGQUKCLiISEAl1EJCQU6CIiIdHqQDezgWb2hpkVmNknZnZLPAcTEZGWiWVzrhrgV+7+gZl1A5aZ2QJ3/zROs4mISAu0+gzd3Te7+wf1H+8CCoDseA0mIiItE5dr6GY2CBgLLG3ia5PNLN/M8rdu3RqPw4mISBNiDnQz6wq8BNzq7uWNv+7us9w9z93z+vTpE+vhRESkGTEFupm1JxLmz7v77PiMJCIirRHLKhcDngAK3P3e+I0kIiKtEcsZ+njgemCCmX1Y/+eiOM0lIiIt1Opli+7+NmBxnEVERGKgO0VFREJCgS4iEhIKdBGRkFCgi4iEhAJdRCQkFOgiIiGhQBcRCQkFuohISCjQRURCQoEuIhISCnQRkZBQoIuIhIQCXUQkJBToIiIhoUAXEQkJBbqISEgo0EVEQkKBLiISEgp0EZGQUKCLiISEAl1EJCQU6CIiIaFAFxEJCQW6iEhIKNBFREIipkA3s0lmVmRmxWY2NV5DiYhIy7U60M2sLfAwcCEwErjWzEbGazAREWmZWM7QvwIUu/sad68C/gxcHp+xRESkpdrF8L3ZwIYGn28Evtr4SWY2GZhc/+k+M/tXDMcMk97AtqCHSBF6LQ7Sa3GQXouDcqN5UiyBbk085oc84D4LmAVgZvnunhfDMUNDr8VBei0O0mtxkF6Lg8wsP5rnxXLJZSMwsMHnA4BNMfw8ERGJQSyB/j5wgpkdb2YdgG8Df4/PWCIi0lKtvuTi7jVm9nNgHtAWeNLdPznCt81q7fFCSK/FQXotDtJrcZBei4Oiei3M/ZDL3iIikoZ0p6iISEgo0EVEQiIpga4tAg4ysyfNrDTT1+Ob2UAze8PMCszsEzO7JeiZgmJmnczsn2b2Uf1r8e9BzxQ0M2trZsvN7B9BzxIkM1trZivM7MNoli4m/Bp6/RYBK4HziSx1fB+41t0/TeiBU5SZnQnsBp5191FBzxMUMzsWONbdPzCzbsAy4IpM/HthZgZ0cffdZtYeeBu4xd3fC3i0wJjZbUAe0N3dLwl6nqCY2Vogz92jusEqGWfo2iKgAXdfDOwIeo6guftmd/+g/uNdQAGRu48zjkfsrv+0ff2fjF2tYGYDgIuBx4OeJd0kI9Cb2iIgI//hStPMbBAwFlga8CiBqb/E8CFQCixw94x9LYD7gV8DdQHPkQocmG9my+q3UTmsZAR6VFsESGYys67AS8Ct7l4e9DxBcfdadz+ZyB3XXzGzjLwcZ2aXAKXuvizoWVLEeHcfR2RX25/VX7JtVjICXVsESJPqrxe/BDzv7rODnicVuHsZsAiYFOwkgRkPXFZ/7fjPwAQzey7YkYLj7pvq/1sK/I3IJexmJSPQtUWAHKL+jcAngAJ3vzfoeYJkZn3MrGf9x1nAeUBhoEMFxN2nufsAdx9EJCsWuvt1AY8VCDPrUr9gADPrAlwAHHZ1XMID3d1rgP1bBBQAL0axRUBomdmfgHeBXDPbaGY/CHqmgIwHridyBvZh/Z+Lgh4qIMcCb5jZx0ROgBa4e0Yv1xMA+gFvm9lHwD+BV9x97uG+Qbf+i4iEhO4UFREJCQW6iEhIKNBFREJCgS4iEhIKdBGRkFCgi4iEhAJdRCQk/j/Pmm3/Gvml7QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.026895761489868 2.9732823371887207\n"
     ]
    }
   ],
   "source": [
    "# 随机初始化参数\n",
    "w = t.rand(1,1, requires_grad=True)\n",
    "b = t.zeros(1,1, requires_grad=True)\n",
    "losses = np.zeros(500)\n",
    "\n",
    "lr =0.005 # 学习率\n",
    "\n",
    "for ii in range(500):\n",
    "    x, y = get_fake_data(batch_size=32)\n",
    "    \n",
    "    # forward：计算loss\n",
    "    y_pred = x.mm(w) + b.expand_as(y)\n",
    "    loss = 0.5 * (y_pred - y) ** 2\n",
    "    loss = loss.sum()\n",
    "    losses[ii] = loss.item()\n",
    "    \n",
    "    # backward：手动计算梯度\n",
    "    loss.backward()\n",
    "    \n",
    "    # 更新参数\n",
    "    w.data.sub_(lr * w.grad.data)\n",
    "    b.data.sub_(lr * b.grad.data)\n",
    "    \n",
    "    # 梯度清零\n",
    "    w.grad.data.zero_()\n",
    "    b.grad.data.zero_()\n",
    "    \n",
    "    if ii%50 ==0:\n",
    "        # 画图\n",
    "        display.clear_output(wait=True)\n",
    "        x = t.arange(0, 6).view(-1, 1).float()\n",
    "        y = x.mm(w.data) + b.data.expand_as(x)\n",
    "        plt.plot(x.numpy(), y.numpy()) # predicted\n",
    "        \n",
    "        x2, y2 = get_fake_data(batch_size=20) \n",
    "        plt.scatter(x2.numpy(), y2.numpy()) # true data\n",
    "        \n",
    "        plt.xlim(0,5)\n",
    "        plt.ylim(0,13)   \n",
    "        plt.show()\n",
    "        plt.pause(0.5)\n",
    "        \n",
    "print(w.item(), b.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-12-09T05:49:38.077058Z",
     "start_time": "2020-12-09T05:49:37.941420Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5.0, 50.0)"
      ]
     },
     "execution_count": 208,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD8CAYAAABuHP8oAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABb2ElEQVR4nO19eZgkRZn++2VWVd89Z8/NMBwDA8MxwHDJITPch466HuiK/FwULxSPVUF3Bd2VxQPQ1RVFPEAFRAVBQK6BAREEG+ZghhmYE+bunrPv6qrK+P2RGZmRkZFZWVdXV3e8z9NPV2XlEREZ8cUbb3zxBTHGoKGhoaFRezCqnQANDQ0NjeKgDbiGhoZGjUIbcA0NDY0ahTbgGhoaGjUKbcA1NDQ0ahTagGtoaGjUKBJxTiKiTQC6AeQAZBlj84loPIDfA5gFYBOA9zPG9lYmmRoaGhoaMgph4AsYY/MYY/Od79cAWMwYmw1gsfNdQ0NDQ2OIUIqEsgjAHc7nOwC8q+TUaGhoaGjEBsVZiUlEGwHsBcAA/IwxdhsR7WOMjRXO2csYG6e49koAVwJAU1PTCXPmzCkpwes6epAwCLMmNpV0Hw0NDY1awcsvv7yLMdYmH4+lgQM4jTG2jYgmAXiCiNbEfTBj7DYAtwHA/PnzWXt7e9xLlVj04+cwtjGFO/7tpJLuo6GhoVErIKI3VcdjSSiMsW3O/w4A9wM4CcBOIprq3HwqgI7yJDUPiKCjt2hoaGjEMOBE1ERELfwzgPMArATwIIDLndMuB/BApRLpSw8AHYBLQ0NDI56EMhnA/UTEz7+LMfYoEf0TwL1EdAWAtwC8r3LJ9GAnQ0NDQ0MjrwFnjG0AcKzi+G4AZ1ciUVGwGfhQP1VDQ0Nj+KHmVmISEZhWwTU0NDRq0IBDM3ANDQ0NoBYNuNbANTQ0NADUoAEHNAPX0NDQAGrQgBO0Bq6hoaEB1KABB2kGrqGhoQHUoAEnQPNvDQ0NDdSiAdcWXENDQwNALRpwrYFraGhoAKhFA641cA0NDQ0AtWrAq50IDQ0NjWGA2jPg0Ct5NDQ0NIAaNOCADieroaGhAdSgAdcSioaGhoaNmjPggJ7E1NDQ0AAKMOBEZBLRUiJ6yPl+PRFtJaJlzt9FlUumLx2agWtoaGgg/qbGAHA1gNUAWoVjtzDGvl/eJEWDAE3BNTQ0NBCTgRPRDAAXA7i9ssmJkxatgWtoaGgA8SWUHwD4CgBLOn4VEa0gol8S0biypiwEekMHDQ0NDRtxdqW/BEAHY+xl6adbARwCYB6A7QBuCrn+SiJqJ6L2zs7OEpOrt1TT0NDQ4IjDwE8D8E4i2gTgHgALiei3jLGdjLEcY8wC8HMAJ6kuZozdxhibzxib39bWVnKC9TIeDQ0NDRt5DThj7FrG2AzG2CwAlwJ4ijH2YSKaKpz2bgArK5RGRZqG6kkaGhoawxeFeKHI+C4RzYM9p7gJwCfKkaB80MGsNDQ0NGwUZMAZY0sALHE+X1aB9MSA9gPX0NDQAGpwJabNwLUJ19DQ0Kg9A17tBGhoaGgME9SeAdcauIaGhgaAWjTgeks1DQ0NDQC1aMC1hqKhoaEBoAYNOKAlFA0NDQ2gBg24DmaloaGhYaP2DDhIuxFqaGhooAYNODQD19DQ0ABQgwbc3tCh2qnQ0NDQqD5qz4DrLdU0NDQ0ANSiAYdeSq+hoaEB1KIBJ2DT7j5s3ddf7aRoaGhoVBW1Z8Cd/wu+t6SaydDQ0NCoOmrOgHMM5uTtOTU0NDRGF2rOgJNeS6+hoaEBoAADTkQmES0looec7+OJ6AkiWuv8H7Jd6VXYsX8Av/77xqFIgoaGhsawQCEM/GoAq4Xv1wBYzBibDWCx873iCPM/+did/8T1f3lNT25qaGiMGsQy4EQ0A8DFAG4XDi8CcIfz+Q4A7yprykKQs9QmfF9fxv49p10MNTQ0RgfiMvAfAPgKAHHmcDJjbDsAOP8nqS4koiuJqJ2I2js7O0tJK4BwA86lcR0rXENDY7QgrwEnoksAdDDGXi7mAYyx2xhj8xlj89va2oq5hQ9ZS+19Qo46rtf4aGhojBbE2ZX+NADvJKKLANQDaCWi3wLYSURTGWPbiWgqgI5KJpQjjIFraGhojDbkZeCMsWsZYzMYY7MAXArgKcbYhwE8COBy57TLATxQsVQKyIRo3J6EoqGhoTE6UIof+I0AziWitQDOdb5XHKEauPNfx0nR0NAYLYgjobhgjC0BsMT5vBvA2eVPUjRCNXC9wEdDQ2OUoeZWYubTwDX/1tDQGC2oOQOezSuhDF1aNDQ0NKqJmjPgoQxcKygaGhqjDDVnwLOCF4p6wlJTcA0NjdGBmjPgIgMXXQq1hKKhoTHaUHMGXPRCEWOCcy8Ubb81NDRGC2rQgHsmejArGHDnv6UpuIaGxihB7RlwQTZJZ3OB30PcxDU0NDRGHGrOgIsa+MbOXvczX8ejGbiGhsZoQc0ZcFFC+dDtL7qfdTRCDQ2N0YaaM+C5PBqJZuAaGhqjBQXFQhkO4Ax8+tgGtDYk3eNaQtHQ0BhtqEEGbhvoiS11yoU8Oly4hobGaEHNGXDOwOtMQ8m2dThZDQ2N0YLaM+DO4p1UwvCxbb6QRzNwDQ2N0YI4e2LWE9FLRLSciFYR0Ted49cT0VYiWub8XVT55HoGui5hwLKCS+m1Bq6hoTFaEGcSMw1gIWOsh4iSAJ4jor86v93CGPt+5ZIXDpuBCwZcT2JqaGiMMuQ14MwWlXucr0nnr+pWUpZQOLT91tDQGC2IpYETkUlEy2DvPP8EY4yvoLmKiFYQ0S+JaFzItVcSUTsRtXd2dpYn1QBMgzQD19DQGNWIZcAZYznG2DwAMwCcRERHAbgVwCEA5gHYDuCmkGtvY4zNZ4zNb2trKznBJ86y+wmDyMe2+UpMPYmpoaExWlCQFwpjbB/sTY0vYIztdAy7BeDnAE4qf/KC+PVHT8KzX14Ag9RsWzNwDQ2N0YI4XihtRDTW+dwA4BwAa4hoqnDauwGsrEgKJTTVJTBzQiMMUkso2g9co1Yw71uP4wM/e6HaydCoYcTxQpkK4A4iMmEb/HsZYw8R0W+IaB7sCc1NAD5RsVQqQEQQ9nPw3Ah1OFmNGsG+vgxe3Lin2snQqGHE8UJZAeA4xfHLKpKimDANNdvWEoqGhsZoQc2txOSQJRTolZgaGhqjDDVuwL3v3qbG2oJraGiMDtSsASfJC8XzA69SgjQ0NDSGGDVrwGU/cA6tgWtoaIwW1LABlxi4818bcA0NjdGCGjbg5NvgmIeT1fZbQ0NjtKB2DbihJRQNDY3Rjdo14KESSnXSo6GhoTHUqGEDrqMRatQutLurRjlQswacJD9wDt0wNGoBOT1U1CgDataAG1LwKh1OVqOWkNUVVaMMqGEDbhvsxas77ANaQtGoIWgGrlEO1LABt/9/7M52DGRyehJTo6agGbhGOVC7BpxbcPgbg9bANWoB2ZyOe6xROmrXgJNgwHOW54WimY1GDUBLKBrlQJwdeeqJ6CUiWk5Eq4jom87x8UT0BBGtdf4rNzWuFAQCjsGcpScxNWoKWkLRKAfiMPA0gIWMsWNhb2B8ARGdAuAaAIsZY7MBLHa+DxlEBj6Y9YajehJToxagGbhGOZDXgDMbPc7XpPPHACwCcIdz/A4A76pEAsNAggHP5JiwJyaws2sA33tsTdFySm86i427esuRTA0NJTJaA9coA2Jp4ERkEtEyAB0AnmCMvQhgMmNsOwA4/yeFXHslEbUTUXtnZ2eZki1JKBID/+K9y/B/T6/H0s17i7r3R375EhZ8f0mJKdTQCIdm4BrlQCwDzhjLMcbmAZgB4CQiOiruAxhjtzHG5jPG5re1tRWZzCAMHwMXDTiQztjfiyU5L79ZnOEHgPNueQb/88jqoq/XGB3QGnh18Mpbe5HO5qqdjLKhIC8Uxtg+AEsAXABgJxFNBQDnf0e5ExcF0Y1wUPRCYcw17tVwKXxjZw9+9uyGIX+uRm1BM/Chx8ZdvXjPT57H9Q++Vu2klA1xvFDaiGis87kBwDkA1gB4EMDlzmmXA3igQmlUQpZQuBcKY0xYlTmUKdLQiA+tgQ899vUNAgBe295V5ZSUD4kY50wFcAcRmbAN/r2MsYeI6AUA9xLRFQDeAvC+CqYzAFlCEffEdDc4RmkWnDHmmyzV0CgXNAMfeozEEs9rwBljKwAcpzi+G8DZlUhUHERNYpJnwUuCxQBT2+9hg47uAfSlc5g1sanaSSkZ5dTAH3l1OxYcPgkNKbNs9xzJGElNumZXYlLEJKYrp8S4z/7+DB5cvk35m/YpH164+u5lOOv7S7Blb1+1k1IyysXAV27dj0//7hX8x59XluV+xWBv7yDOu+UZbOjsyX+yRllRswbct5An54+FYhj8c/77fOne5fjc3UuxriNY+SzG8PKbezGQGTmz1rWM3b1pAMBjq3aW7Z73vPQWXt2yv2z3i4tyaeD9Tt3ctLt66xYeW7UDb+zswU+fWV+1NMTBSORjNWzAvc+DWctlyxZjwrL6/G9s275+AFAa6a17+/Evtz6Pr933ahlSrFEq2lrqAJQ3ENQ1972Kd/z4ubLdLy7KxcCTpt2ERRlxqMFzQjUiToykaa2aNeCm4ZdQuK22mPeCSm0i3QNZAMCrW4eeoRWCHy1ei/f99PlqJ6PiEN9xraNcGnjKMeDDwatl+BvGEVBxJMTxQhmWICkWite4CwstG6fSDXct/KYn3qh2EoYUw/19xEG5GDiXCweraMBHwOuoWdQsAxcllEzOcl0GGfOMe6kVixsKXUGHB9xOukzGr5qhh8vFmC2rvPcrBrztDX8GbqNGkhkLNWzA/SsxLaFxc+NeKsvJ5DxdXaM6eH79Ljy9xl7kyw1Fuexuxqqe0SsXA+d1s5oa+FCAMYa/LN9WUj5HYjOuYQPufR7MWq68JS7kKURnVL1cXlni3kXvBlR+/PSZDfjBk7ZEpJLJSkEmV733VS4NnJdFNfPioXLc9tm1u/DZu5fipideBwA8unIHPn5ne1H3GkmL80aEBi5KKJawerJUljOYy7n3jIORMLk23JDJWshJUlbZDHgVWWv5GLj9v5p5GQrewsnU2p22u+8nf/ty5R9aA6hhBu6fxOQVmTEmMPD4lVq17H4w63QKMW+jpZbyI2tZblRJbsjLZsCrKKGUyxWSdwRVncR0/leS2LbW21yzqz9T9D1GYuusWQNuCinP5JgrX4huhIWwHNW5vFHElUZ0fIvyY1B4t7x8y2WrRoKEwsummgaco5LCRMJp8Ny1txjwZjxyBJQaNuAkTWLy5lCohBJl7NPO4p74GnjMEzViI5vzL9ICyjfXUE3ZgXuNlMpaeb2tat0bgofzd9894GfghdSFkThCrlkDLkoofems54UiTGKWj4HHvEeVK0g13eIqhWyOCcy7zBJKFVkrZ/9Jo7QmOBxe+VBIKPzdywy8kDY+EttHDRtw7/NLG/e4PTETohEWMkxVGV8+cRJ/ErO6FaTaHUglkLG8RVojSULhnUeJ9rvqdQ4QpYnKWXBufLvTfgNeiE3m544gJ5RaNuDeW9i2fwBv7OwG4I+FUljvHDzmGfC496iyAR+BDCOTswKTlyODgZdpIc8wMOAcFWXgIfksJP+VIDhv/97T+N2Lb5b9vnERZ0eeA4joaSJaTUSriOhq5/j1RLSViJY5fxdVPrliuvzfB5x9MLM55jLvOAycG3uVx4rrB16EG2E1fMJHogHP5hh6BrJYs6NL0HtHggEvz6Kk4fDOh6Kuh2WzkEdzY1/OkcKbu/vw9furF8o3jh94FsCXGGOvEFELgJeJ6Annt1sYY9+vXPLCYYR09/ct3ep+LoQRq3pyVwMv4h45iyExxLtBjEgJJcewu3cQF/zgb5g5vhFA+fJZTc8Nd0VhiVkZDq/c1cAr+IywtlwIAy+lo+noGsALG3Zj0bzpZblfuRBnR57tALY7n7uJaDWA6dFXVR5iNMIwFKSBR0ooMRm48LysxZAY4g1ScsNiNV55IY6MCpW08t67qm6EhdWtMAwHBs5RyRWOYj7FdlaQhFJCf/2vt7+ItR09WLy6A0s378XfvrJwWJR9QRo4Ec2Cvb3ai86hq4hoBRH9kojGhVxzJRG1E1F7Z2dnaakVEMN+I1fAQg3VuWluMGK+KPG0agzPRyQDF1z90lnHrXMkSCh8kViJeRkOGvhQJEGs2/1C7P5C1mJZ3mxrwVjv7Db04PJt2LynP5CmaiG2ASeiZgB/AvB5xlgXgFsBHAJgHmyGfpPqOsbYbYyx+Yyx+W1tbaWn2EtP3nNKZuAluBFWg90NB0bw+o5ubHU2ySgHMkKe+DxHufI5HCYxS81JoeGTK4GheKpIovoGBQM+RBKKqspVcSGvi1gGnIiSsI337xhj9wEAY2wnYyzHGLMA/BzASZVLZhBhGriIqIb+1u4+dHanvXMVLzedKV5CqcYy7eFgwM//wbM47canynY/cck5Z+DlyuZgmTrZ0258Cp+7e2mBz/bIQbkMS7VeP0//UHmh9A16roRDJaGoUEiojkohjhcKAfgFgNWMsZuF41OF094NYEinYuNIKFEM/MzvPY2TbnjS/a6SSUqZxBytDLycyFlMaaDK5a6ZVayGXNfRXfB9tu7rD90YOwwi+y+FOIvvvNrvv5J+4GLeetMiA1efv6snHTjmeaGUB8PAfsdi4KcBuAzAQsll8LtE9CoRrQCwAMAXKplQGXEYeL6Gbm/+YH9WGftBh/HFbRjiadqAF4Z72zfjydf8mxWHSRzl9gNnzN4T9aEV23DOzc/i8VU7ynL/6GcXNxEnQ7y2lPvs6knj/qVbir4+Lu54fhMeXbm94OvErIkMXDV6WdfRjfn//SR+88ImvL6j2+2Uyz1fMBw08DheKM9B3Wk9Uv7kxAc3vEThDKYQDVxl7Hkji6uV5kqUULoGMugfzGFya33B1wLlC5BULEphxl/54woAwKYbL3aPheWnEhLKnP98FJ8/ZzYAYOXW/Thv7hQ7DTkL6zt7cfiUlvI81IGPgZdwH9l1tVh87I52LNu8D6cf2uZuHh0HO7sG3N3o40go1z24CoD/PceBj4EPRjPwXT2DAIBv/uU1tw5tuvHiog14mMRVExLKcAV3IxRjSZx75GRMG+MZv4JioUQupY8X/pOVKKEs/P4zOPmGxQVfx7G3bxDn3fKMuyp1qNEneAeUA2FlXi7mIwezakjafp+il8N3Hl2D83/wLDbt6i36ORs6e3DhD/+GfX2D3rOFvJXEwIUslNKBb9lrTzwXqsd/9q6lrsHMh1K0fp8GLiynV9UF3vbk8uBlVahW/4Hb/qE8Pgzsd+0acC6hiItl6pMmLn/bLPd7nB4yavcePmkGxIubUaoboUq3KwRPvrYTb+zswf89va6k+xSL3nTxoT5VCCtzxhiuvLMds655uMT7SwY8FTTg7W/uBQDs7i3+3fz4qXVYvb0Li1d3CM8WvUe8c9/a3YfLfvFi7LIUDVgpI6BiO5GugfjxufeXEMvbCmPgedqtiGI7/pc27om8X5z5uEqhhg24/T8pBAYn+Bf4FBOprF+oHOL+e3H24hOfVw05gxuelFAm7Zv2YMveviF5fk8eo7N2ZzfueH5T7PuFauAW8Likl4dhy94+nHzDk9i8J1gG8juqdxg4d1NbuXV/7LRGgbtCimQjjIF/57E1+NvaXVi8xjP2URBZbSkjEz7ayRRYb8W5qHx2TPT6kpHJWfjC75dhY8hIJ8wLJWorRBmut0yZpjH5wrlqbtFWswacIyk0CiIgIRjwQmSMnMVw90tv4YhvPOoeS4uLSHL55QGxIVbDx5h3PsmE91rf+9MXcPp3nh6S5/elo8vo639eieseXIXXtnXFul/Y+yvEUN33ylbs7ErjD+2bA7/JDZ1LKAOZHJ5+vQOX/Og5LH1rH4DSPEW4cUwIcp/47NK8ULzPxTLwnMVcj6tCdwoSoynms2MdEQb8lTf34v6lW/GVPy5X/u5j4OloP/B0iAEvllOF5SvndgjVQ80acP7exEZhEMEU2GchDd1iDPe/stV3bM0OT0sO69XXdXTjvle2uPfId345sbsn7eso+hQMfCiRj4HPmmDHMnlg+dbI8zjCJoIL0VJ547p/2VbMuuZh34YAcifLG+pAxgpo3tc9uKpoiYvLJSIDF9l/ubxQugayBUkaHO+59XkvGFwJDDzfpZyBj2lIBn5z46OH1F1xdJvPDzxUQinSgvMYPGH3i+MRVynUrAHn70JsFAQgGUNCURmArMXQlwk3QGEG+Zybn8UX713uSxMQXonKhYFMDm//3hL88WXP9WvAYeB1ieq8Vq7bhj2/uc5uuHH3NQxl4AU0RN62+PJnPlkHBI1VTpDR5Ca5alsX/u3X/yzKQPIwDUmTkM1Z+EP7Zgxkoj0p4poE0YCdc/MzOOb6xwtO3/LN+9zPhU6+k8+AR1+7p9ee7GxtCDq/8c46EWLAxTLKV3YqBj6YtYpecHTAuGgDHvayBjI5zLrmYfzM8dKpBGrWgHP4emzya+BhbEJlAHIWi5QA4kSuE+/LGU2l0DWQQU86i+37B9xjrgauMKD7+zN4ft2uiqap12FGfDJQBm/gcQ1wuB94/DTJ+qRvnkIyVjx9/SHeNCu27Mf7f/pC/Ifz5zjPNA0Dd77wJr78xxW+jkRFKOJmsdwx6AuV/sTSzTeQ4E4FKg066+5QpLaG/N2YBvnIlNxpdHanfUvtOfoHc0VLKKqOybK8naLC+gM+aXv7cxuLe3AM1KwB57vIi5q3QeRj5GHR+VSG3bKY8sVzxJFExIZYaQbO9e60YGx4+lUSyid/8zI+dPuLroSwobNHGbPEshi+/fBreGt34ROfXJtsSJpo37QH7//pC75y4xU+7jA9zkKeQofF4rPlAGb8cQOZXOjElCirqbBiy76AG2fG1cBJ6c0i2odCB+PlXi9WaHmK9jbftbx8maJ74tp7PgmlPmH4yNR3/rrGLd+BTA4nfvtJ3PjXNYHrewezRUsoKgOeY8w9bhBh7c5u3KuYZwEqq5HXrAHnbc80yFvUA5vlcIQZCpVhyFrMp63JKNQLJV0CA4/DqrixFoeT3KirGPjqHV2+NC686RllzJLXtnfh53/biM/es7TgdPPyq0+auOqupXhp0x5s3x+ULKLy19E1gO88ugY5i0Us5Cl+slg02kE/YY+Bxx1my+z5nT/+O8675Vnfsay7eQPz6aXc+EVJD/96+z8w15lYV7lOljuAVaGLU4wCJBR3R6WIwHFhMfT5fFZDyvRJJIvXdOCBZXYYg7DJS8BuL1aREoqqGor1kwi48Id/cxejcQzFQs2aNeC8Fycil4UTyRq4+oVmfcuY+X81A//ue48BkN+AM8YkDbx4Ax6HofK0is953WF+pkF4YNlWnz7OjVM+FlLKJgd8IsogYEeXLe2Iw2UrhIGLBv2rf1qBW5esx4sbd0e6EXrPjE6vPMEkvnu5LLiRUGngYYjD6rJC2Yv3rXMCxkfd4u/rdrt+zyrXSdXze9JZ5QhwQ2dP3gVJhWrghUxiRu2olM0zicnrSH3SDKwPiBNmuG8wW7RBZYwF9h+wmF9CySral2ejintuHNSsAT/AmRn+5NsPdguXQEoNfOlbe/HPTZ4zvujdIO54rjKc3LUsncdQWCxaQtmxfwCzrnnYl44wxGFB/QoGzpGzGK6+Zxn+/Q+eSxZPWr7Oga9OTBWxm5CKgfnKOkQDF8/x3NlY6EKeQoKGyY0nKviTTwOP2erieDrx98mYP0F8pKTUwCPu+18PvYbvPLoGu3rSSqN51HWP4Z0/+nvg+MKbnsFZ31+SJ63ho9a9vcEVl2Ix5RsN8PJWPYKXUSJEA+fNryFpujGKOMKIgYjedK5oP3mLBdNlB1rzJBQOsd17Br5yFrxmDXhrfRKbbrwYi+ZNd10JDcM/BOMTfO/+yfN4nzD55NvdgxuVkJfLF3fkY+A5i/nukc5YyFkM97z0FjI5Cy9u3A0AuPOF/BugxmPgtlyhmixV5YXnMx9jzefOFQWXYQnHvnTvctd9LKdgKYCaFecsFuqTXIiEIjedbJQB54Ygx+J7gcQYsPD8BRm4Y8DF9MboOH7x3EbcumQ95v/3k6GyxetFhlMIK88v/H4ZjvuvJwLHReOVbzTC06pe9RzthcLrdH3SDLTFsHolom8wG7knZs5iysVePN3yvJJlCfVWuJ3YHl0Drhl4NDzWTT4NfNOuXrXerTIYIUyuIaYBtxQSyr3tm3HNfa/iV3/f6DbMOD6/cYax3FNCNVRW5YUfyXfvjGIyaUNnj0/LFmFZzB0NqBrSss378D9/XR36OyD5RDvF3J3OurugBJ4pXK5aOZjO5vDbf7wJy2KRDDzMjTDHgteFIc5oKRty37qkXcalxUIpr9AaZgQfWmFHEJQ7VXEhT76keDJDsMwG3DUM6oJnTtmlpElM8bliW5eJfN9gLrKsfvTUWpzx3addI84Yw/89vQ67e9KwrOD+tuIkpviLOCJ2NfLQp5aOOJsaD3uIGnhCklDeVHhTiC86Jw6bFWhI2TU0DgMXK0g6m8NORwfuGciCxtjH40w6xTEK3iRm8FxVXjz2E31v3iGIBnzhTc8AUEeQu+GR1bj9uY1Y9+0L3WfIZcVZWi5ktCMaBZ6+qA0SfFEfFe/lJ0+vxw8Xr0VjygywLd+7D2jg3vG4w954DJxLKP77xtHA86HcIU3DZCvTIOQshnTW8rFkkYHnq9viCEcGr8ehDNxiMImQNMndjs69r0KaG9uYcv3O+Tn8Z1Xn3L5pLwBgXWcPpo1twG9e2ITvPfY6lr61FxYLjkhzghuhYZBbPqIB9xi4llAiwRm4QUGtaoOCxfkZn/158141w2xI2n1cvsk9sUcGbAbOh4V1SdOt6IUMuaPgGnAFA1ctWeZZHsxG39vzJY9X6f6ywvYA2Ly33y1XedTDX0kuF2xogCRrxLBHoqE46/tLfA0V8AIs7ekdDDTWQUVnwSFO9MZtc3EMaMbNt/94c13C91zffSMmen3Hy2C//fNG6grKQ1ZETc7nKwueJ9UKW9VcjnxvwyCkEmZgPkrlniqv9hQ1axXGNtrn7+4ZxI+eWovr//IaAHtC2GIsYMDlSUxehioJpZKIsyPPAUT0NBGtJqJVRHS1c3w8ET1BRGud/+MqntoQJIRJTHmoo2KjPgnFealv7lbPzvNFKXm9UCx5IU/OdSWsSxjg71/lAxuVvjD0Oxq4yl1RZcBZTAbOO4a4GviB45sA2B0lNzJyWfHKHTqJKQZ2ilHp5VO2Sf7sPO1ZiwU8D6IZuPc9LmfKKibE5c/8nBxjvue31IcPgGV5J4xAlENCqRfcTsOMDg/bLEt2BXmh8DoYwcDDnm85DDxlGkENXFGvWqWytZhXVio73uoY/C17+/CKE/8GsG2KzcCDk5giw+bebwOKScxKIk4rzQL4EmPsCACnAPgMER0J4BoAixljswEsdr5XBabpSSiiBg6oK4uv0eW4AVdPYMTVwHMKDZxX9rqEIWjgkbcJpC8MkQy8ayBwjFdaeYh8xa//6XM37I9YDKQC9wba0NnrMSzpGd7ow8+Ueh13t0KjOMoNQ3bx4h16JmsFWKH4HsM0cCD+xNOrW/Yr7y2uKeB18M9Lt+LHQqjf1nrbaISt9BMRHqCpDAY86a2cDZNQeIA0mTCIRZ/XD1x4/7LcwutxWN3PWfZ7rksYAS8Ur25719YnTdQnvTpsWV77VKWTv7vNe/p9spy9YUwIA3e9UEQGrtDAqzmJyRjbzhh7xfncDWA1gOkAFgG4wzntDgDvqlAa88L1QhF8wjnECpHN2Z4hGQUDD4NrwPNJKNIQLZ2xfI2Op4pX3Dtf2IT5/x2c1bfTHF9CkRtUY8rEvr5gvA5XA5fysXhNh8/dUBXRMApNdXb5rO/scctSLiteuWVf2bnXPYZFP/67/33E6LzkBij7evPGlhFYEodqZah7X+F7nA2Pd/WkccUd7e53kZ2KYYl5vp+SQsS6EoriUXIdKHeAJhG+IFsh9TxMQilEAxfbmpxubvhUj2eMoWsgAyNkElM1OV6XNN22C3ANnLmfZfQM2B1uZ0/aZzOI7PNlbd7yjbjJrXNpn4RiufeoFArSwIloFoDjALwIYDJjbDtgG3kAk0KuuZKI2omovbOzs8TkqiEyMJmNicbh0K//FR/99T99lVSsC/K1gOcpEM8LxX5WQ9JEOptzK3s6awmVxz7/Gw+swq6eQeUQOJ6EovYDN4lCJjGde+dp8PxaXhL5DAQv39e2d4UuFjIkDxzx9zU7uv2dbAyDJLc/mbVxY5PNWYGyFA1zMBaK93kgIqwChxw7x8/ABQMeYhSb67kBD0o36zt7cNeLbynvLSLORiP50Jjy5Iaw8k+ESChUgBuhWAyBDsqVUIL5vPWZ9fjjy1vQNZBVTmKqNPCUafgMeE5gzKpkdqdt0pPNWYHO22JB75icqIGT1wmK5cPr1+Y9/dixPzgqLgdiG3AiagbwJwCfZ4zFC+gMgDF2G2NsPmNsfltbWzFpzAvfSkypoGUD9+wbnUo/cMBmr6p7J02KxcD5fflyXx6nZDBnecvI5SG9cll/DAklozbghkGRcVgyOUvJlPh9uOHhlS/f5BI3Tmt2dIeWkeuFEmLgfStj40goUvplI+Yy8JwVKMsoBi7eN8wrCbD9sGd//ZEAs0qHGPCwTRK4Bq4irr/6+yZ87f5XlfcWUY648y31CbfuF87Avc95NfCIjprXM1UH8sir3ibIajfCYL1qa0mhPiUycK+cVfWfM/CsFVx/oGLg4oib4HVwYZOYp/xP8VslRiGWASeiJGzj/TvG2H3O4Z1ENNX5fSqAeFuIVAD+lZj+LKliVIsNSixklQE3DfXEiYycMGFmM3ALA841g1mPCcquTGoDHoeBOwt5pHQlDIqMhJjNqVec8olAuSFFGTLxvMGshbU71X7bOcvC0693KJcbi/cA8jNKg4KdoGzEeH3IKPIqnpu1LMw/cJzrsSAamKh8/9dDryGTC06Q+uLSCKGJw5hpi6OBx3EtveHh1crjUQY87gSnxRiOmjYGALBTMQHeN5hVSgQy8q/E9D7LRpK3A15WK7fux1V3vYI3d/fCFFevmsGFPKpFalNaG/wSiuW5+aoZeNZNl3gfAoGxIDG0GHPbtBhET6WBVxJxvFAIwC8ArGaM3Sz89CCAy53PlwN4oPzJi4dEhBsh71lFiJXH8hnwoFcAEdm9ft5YKF5Fshl4zo2PPZj1KgWv5HxWX3XfQtwIA3JFng36spalNCi8AbkM3DFm+Ri4WNn3hcT5vuOFN/HRX/0TLzv7SwYZuHcPVYREEQnTiPQsEe+fyQXzKjPwproErjj9ICcdAgOPI6FI8eNF1t0/mN+drKU+XAOXEbbFWlSHp5rgVsGygKTjNnrrkvV4fNUO97c/vrwFR37jMWx2tuWTR3eFRIb0r6BVj8L4PZ5e04GHVmzH9x573VenVW2Rv37x+VPH1PslFEuUUMIZuL0CWJC0HMKg9APnDFywOz4/8DL76KsQh4GfBuAyAAuJaJnzdxGAGwGcS0RrAZzrfK8KTEFCET8DXoxqEWLlERmwOBsvIo4BFzWxxpSJdMZyQ7eKhkSMawyEGfDgsbU7u/G2/1nsepiEhb4188yYZHJMydp4OvolBp4vrrlY2eOMUux7+88rRMdNOgsmoq7nac/mWKAz9DNwhoThxc8R60KccMAPr9ju+y7WtajIlhxNKYUGXuCEVxQDjxuT3mLMXVQE2KtnOZ50Amjxe8kSil+OjH6O7Gb5wLKteHTldt9vsp49kLH8DDxCQhEZb1trnS8uvcW8EZMqnXyknskx5SSpyoBz8ueTUESCUO5YvwrE8UJ5jjFGjLFjGGPznL9HGGO7GWNnM8ZmO//3VDy1IeCFR8JQhuuu3QoGfv9Sz21ObACphBEYKgH2irl8DVolofAFJoNZy5VtuO3iPbZK21QNvX77jzexbf8AHlxuL5wJY4iqiVgRz63dhd8Jk2Mc3Pjy8uCdSF4NXDDGcfVYOXv8uoMmNuW9NpkwghJK1sJHf/USHli21Xc/u+P0pyktMXDTILeuiOmPw8B/9NQ633eV50kU+LsSs1MoaYuaL8n37jhshunVG5HIyPVJbgdhW8Pt7BrAZ373ijsKBfyMNJOzcPU9y/DJ377i+82V5Hg9tCw/A1e0T5e9CwZzXGPKlw9x4Y0s9WRzlm/kKXf6oRKKO4lJkW6ElcSIWEovsm7XmMOWVHoVGvhjq7ywnGIZp0xCwjCQkTYwbqpLoCfPhr2iX2hDykTfYM4z4DnLNYguAzcjGLiiUU4d2wDAC9AVtv1bPgP++/bNSh9vng6ZCeWXUPyNMg7k/PGVk/LiCxUShhHQdjM5C0+/3omnX+/ExUdPdd0o01kLDREaeM6yY1zwIvNJKDGNn4jeAg24Kh54ocPuqJW1cfKwatt+bN83gNmTWtxj4pZ4AQOeUbNfwDZ0OYvht/94Ey+s341HV+3AGbMn4tKTZtrnCmUil488PzIozB/JDFwGr09ivTqkrUlyIxTikUtlLG6SnLX8I1QiCpVQxPji/POwW4lZC+CsWwwnazGGhGHk3WhXRCphKAPKt9QlfJvhqiC+0MaU6Qv1mc5aQu/vpLlADXxCUwoAXHckmSEe3NaEX/2/E/MacEA9ccqXJ6uGsFHIWpY77I8bA92y/CyIhykNk7BEJE0KMHgxP//+h+X49fObANgGLJ8GbhqGb9KTo7+IDTn6fYt3nA47ohFzFzzRnhQ67I6WUPIb8Iv/9zl0p7P+IFtRBlx6x3JguPte2YLrHlyFRx0dXTxfNNpy2mQXVM70B7OWLw0q8uGOGp1rH//CmWipTwY0cP54mR9xF0KeH/8kploD9y2lJ7G9iJ1B6R5C+TAiDLifgTuNArZhlyWUj59xUOh9kqahXELeUp9QdgSypicGnRdhT2I6BtxZSs+HZIO5YCNTsTfeyLft78d7fvJ37Orxx/+4/SPzsWDOpFgGXAVu2ET9GMjP4jI5hnpHPy2EgYtlt6fXbkBxDLjhMCIxmyKD+rOzOwtg69BRS9K5Bs4Nqdjg4soPIvoUDFwV9wMAll93XlkYeCkauNiZiQty6oT3IC+SCvMA4Z+fXG2PbhPCHM9n7noFd734lq8zk8M9iLH7Z13zMN5wPJoGc5KEkgjWkYw0AcqfLWrgjIleKPb/Pb2D+Our29223VyXCCz0s89XxQP3nmeQNy8jEivNwGPCi4Xi1xUTBgUM7+TW+tD7pEwjEIYSsBdcqLR0sfFc8qPn3OX4sjui7UbIJRT7mFmgBs4Nwc79A75YDRyc0eebxAwDb5jecueYGnjOcoe1cScjc5Y/j3v7OAPPXx35oiixo1LJZIC9vF82cLKEYhoEPugSfyvZgDtlEVYmYxqSrnEUzyi00WdzwVjVHGF54KOfnULIBbE8RaMsGy5ZAxfTyxiwapu9RIQvUupJZ/Hwiu342v2v+jqnHVJ4Yj5X0eW0s5c22lNqtoTinaeSULx5G27A7XPqfQzcyxdPxtX3LMWnfveK6/46piEZWDvAl9Kr/MDFYFY8DeJEtjbgMeFFIyQfg06YQQlFtePH4ZNt/S8pxCwREcbAZUPLYyaLQ7eGpOlbyCOzhLheKK42GMK4uKaez40wDGEMPP8kJnOH3HErbM6yfGXHWUtDDAbeP5iFxfwrAMNkso7utLtfIkc6Y2FdRze6BzLIWhYSBrll5pNQBGP83hNmxMiV3/OEGyRVuFtucEnBwAud+BrMWcqJd8Bft0T2yz+KQcDEai/KInJ9ivJCyTHmdqa8fr8ubAItnrttn9d5MBYMeSDmQexcVHnNSPIfbwtifepJZ/AHJ+YPL2++0chbTgzwcU1JeyGPkBZbQgnaDXHOiwQG7pu01QY8HjwvFG9i6IQDxyFhELok32RT6kknt9Zh7vRWAECdaSgZbHNdEt0DmcDstdw4+T6QYsWZOqbe16vzyhbtRqhg4Dm/Ni2DV7CwLanyYdAdIcgauGfINnT24Iv3LvMx1UyOueEG4iJrMZ/Wy1ldHAmlL2NvTmsqDLgsH01sTgWu7xrI4Jybn8XR1z+OnV3pUC8U0Ye6raUuTrawZrtnrDIuA/e/388tPBTPfXUBAFEDF4xroQzcskLj1vg8bnydhH18h8DA5W3BuD9+kIFb6OgacD1+cpKEwidyeb1+det+AMD0sQ0BLxWOQYW/vvibT95R5JVP5PJRqieheOf+/G8bfekEvAiEbzkj53GNKWelsndv5pwvN6uHVmzDM6/boUFEDbzQiexSMSIMOB9WEexG8fDnTsevPnoiEkZwwitpEP74yVPd73bUMttwJE1DqSG31CeQybEA+wjTNxuEBUETm+tsCcXyN2h3ElMx2aV68ZzRicZdHDq7o5ASGbjoq/38+l24+6XN7jn/9ut/4r5Xtvo2xs1ZlquBx4VlMV/Z8XKNMuBJkzCxOYUL5k6BHK9bZn0cJx88IXAf0cjya7iB8C/k8dIXN7Tu4jUdONV5ZtiI6YiprZjkyHg8uaLBUE18fWbBIbjslAOVz8xkmVuXZKj2ZxQ/b98vGnDvutue3YDTbnwKa3d2K7xQcvjSH5bj6nuWYcvePncRje2J4dUjbsh4RzBzfKMvDaIk2RexX2VgElNhwJ9cvRPrOnqCGnhIfeKP4tEgNzqhpMc2pgIjTj63JY/M735pM150ZB6CN2rui5BQwsIUlIIRYcD5ijZeyHOnjUFrfVK5u4dpEObPGo+b338sALtH5wYolTCgagstgp4nIkzfbBAYaUPK9C2lf2NnD558bafr7eIaTgVDUj1L/E1kI7zSFrEXsT8dgoTyoZ+/iNe2e2FvNjlMJWkaGMjk8OelW5EtloELlXswhgE/pK0Z7f9xLqaNbXBXvX70tFkAvPciG9q501oD9+mW3qHtheKkI2cpV9Q118XvoE471DbgGXfE5a8jojHygnx5xlbFREVPGRmZnBW6DVlY3BdOEMRVyuL9+QT5A8u2BSYx+wZzbl22jaaFC+ZOwVmHtfm8cIIrhG3vD046RK24dzAb6n0jM/CUqX4X3/zLqoAGPnVMg3J+gDNwHkJho0NIxjUmlStEGQtO5srwQiSHM/C4XlqFYEQYcB6WU4ZKTuCNnFdYMW5wMlRCse8vRxQL61HFJfmphIG0MIkJAB+7sz2ggfsamKIyi4aVQ2R3piuhFPdK5VgUUcO/HGP47qOv4/O/X4YNu3p9q/jiwGJ+Vy3u6SJO/n7i7Qcrr+VbV1mWx8Y4A5c72HGNQQlFhu0H7kkovBPp9xnwpPJaFTi7zuYY9vdnAhKK2Mnwqnb73zbg8P94FHt6B5XlbgoLRWQM5qzQbcjCXPi4sRQZusgweX3/8dPr8Pz6Xb57dg9kcXCbveBq9fZueyLYKcO9ijDGHJksQ9ayXNIhRnLsG8yF1jd79OrlQ8XAAftdyxr42UdMwvPXLgxMjm/a3Yd1HT3urlOd3WkQqTfYyFp2TPmogS33fwdkDTx8EVm5MDIMuFPw8vBH5dPNj3FDV5cw3EZrGuqetsmp0Jf86Dnf8d6QxT1iFDS+9FesoBOb6wJeKKI++Mpbe7Guwz/U5x2AeB+xUbsx0Yt8o2ELeVSwLIYdXd4EWBzvEREyA+eVnjegtpY6fPhktWTAF03kGHMjRYa9h6gdbzhMSULheRHT1xzjPhxtzbZefucLb+LYbz6ON6Td4cV3xvfHfNxZrr6vTx1eOGGGG/BsjhU8icnrkGhQDAJ+eOk8AH7dXt6pvTudcYNwfefRNVjfaQebMgzCvj6/a6svLTkLOeaFZxYXovWms5AX14h5EElLWF7HN6UCGjgRYWJznbJNn3PzM778N9cllFJZJscCXk8yxMBWHd1p/OI5W2+X+V2c8AyFYkQY8BbHwMrDYxUbdaUG5yeRgVtMrSGLS7z/vHQrnlpjN7iwndobBSmgzolkKFbC8U3JwEIe0WA8tGI7zrn5Wd89VQzlQ84KNzs/5PtfKFwGrojsJuO+pVvxyKtewKNCGThj/vv3pLMwyNMsbbctdT5MIpfxmESRi7W4xhkFMRaKyMBFxOkIOMY3pWAahF09tofDakGCAvykQn5VquiJgL+TkZHJWaEaPTdQPekstgh7vqpW2RpEWDRvuh3HRzBssqTQ1Z9VRn80KNqNNGtZsCwv5kpvTAaetfyxScIYeG/ak2HkNhDmWit2cK31SeWIPWtZAa8nGRbzS5v/9ZC9n2aAgRexOCwfRsRSetfnVPLVVjJw16jbv9UlTDcEbc5Z2CHjsMkt+JfjZ+C5dZ34/O+XAQDWfvvCwF6MHI0SA8/kLGQsC811CRw5tRXrO3sw0WFq3iRmdB7lCv6VCw73LdTwOqYiJRTuRhiy8bCIW5es931XeQbkg5j2vsEcGpKmKwXYsSf89+QNyCByG4vhMPBQA96Q34AbRK6UkbWYMqRwS4hEp8L4phQShuhWJo0KhfcjkwVV9EQATkdVuAEfzFo45YbFPm8TwDM2fgau9mKSJ2G7BzIB6TCqg3HTmWUwjaDsxT9H1TfRpVNV1w6b3IzuAW/RlpyHsKSJ+R/ToJ4zszXwaAmFr/JsdEJoADYJ4R44queVCyOCgXONMo7PtxuDxKmEdUnDnfizGAvVTeuTho9FX3vfq9i6L7jLRkPSdFeyGeRFMsxZDBOaUzjt0InY3TvoDhnTWQsPLt+GPSHDzxfW78aLG3YHdPGkYfjyx41BqZOYqshu+RDH/U+GWJl70lk0pEx3eGwx5obblWGQsBjKCfUrzvxPG+Mt1IrDnBngY+AJwwhMfBUioXADzhGokxEMPJ21whm4dPL5cyc7aQ4f3vdncgHjDXhkQWSE3ADn87jpHsgGNqjgEkoUXt/Zjde2d3kauBh6VxHyQES7E4YYUE9ittQn0Z3OuAuzZLYcOn8g1MGxjWoGnsnZDDyqg+K2RFSBHly+zRdzCdASSihawhi4wghww8CHj/UJ0618FmMYr/AdBuyKvVcwso+u3KGMXd2QMt1KmjANdzOIbM5m95Nbbea9s8seYm/Z24fP3b0UH7+zPXAvAPjgz/+BD9z2j4BnimmQcoSRT0JRuVYRqRbyxGcLYQz84c+djtmTmpXpExtPbzqLuoTpvi+G4OiJfzOkTispLdYSl0/HkVCygpdD1jGGsldN2CS5Co0p08fk5FWifiPhz+PH72x3PSIAoMnJS8KkgAwg7jqkeuUG2duyqeCuss3mfOeL9w1D90A2sP7BCGHgh01uDhzzDLhXLgN5DLgIlYTS4qyUzliWsv6HGd84BjxrMaUfuAguKfX71kz0Bs7TDDwEzWEaeMQkJt/urD7prb7MWV7QqMB1gk/57EnN6ElnsXlPXyCCnmmQ59ViEJLOJCYf6vJhPV9gxF221nWoGxuHzMySJilZqueNEsz7CQeOw9lHBLcubU4lAv7ohTDwMANenzQDxpCzW5GNWMw2vPzdqCQUDrEx8p1QRDlG9ACKw8AHs54B53E35BFFSwFeKCTJHb2DERKK9Ir29A76DNnkMdxfnCAXBy/HrOVn4D+8dB5ufM/RGNOQxKtb/EN4DjdglMjAeb3JM4QbFEKvenkipYE7YFwjXr3+PLx/vreSlWvgFvM6KDHYm4gmSc761f87UTmJ2VxnG/BcTi2BhunXYh0c05BSSiiZrL2wh4hw0MQmnHzQePe3K888GAeMb3A7gpOE3xoUUlwlNPC8BpyIfklEHUS0Ujh2PRFtlTZ4qBq4Ae9JS6suFS/T8/P1fI+9BRXhEor4cg+bYi+937av352RF+/PKyln4HwTBb49G+D11s++Eb7Rs2jUg7qjOnY5z7OKad/y/nn41FmHBI431pkKBl66hGJPEEoGPMENuD8/DUnTfTdyfGrA0zHFV2oaQUMvNpw40g5/L+5nCnrVxJFQ/vjJU3Hfp98GwG8EAwxc+K0pD7PnHi1ineIQmaj428zxjbj0pJmoS5hK+QQAlrzeiY/d8U+8sGG3e4yXLy/PsPgqALC7N+0rI1kD5++uPmmipT7pT6twHSczAxn1JKZY7uceORkL5kwKYeBJJzSC2oBzfOGcw7BwjkdgRH2/LmEor+XnGER4+t/Pwk3O+hEAOHRSM04/tM1l4BceNQUfPmUmGlOmci6ltaH8U45xGPivAVygOH6LuMFDeZNVGHiAqktPnOk7ziujWJjcEJ9x2EQAwMXHTHWHp1ynVkE0KHOc2Ckd3enAizIcXZZfwz/3Z3K2QU/4DXgUPvTzf7if5QqeMINMkT8f8EeU46hLGkq23JRKoDudRWd3OnRn+SiEMXBDMfkWacC50YiY9fdJKEQBQ6NqOFEYzHnDYy6hyCtL43j2HDihCcfPHAfAz7LlUA5ieeTbwIKXlWkQPnzKgT4mK3ZcYj2oExalheHbj6zGk6v9W7SZrgbuMfGwe+zpHfSNdGQDzsuvzm0HYmcjGHCH/ITtLiWSIx5uQJWmVkdCseO7q9JsX5tMkM+DZuVWz0PI3lk+eC2vp/y1iXWhLmH4vG8SpoHW+iQGs0Fvpp9++HgcM2OsMp+lIK8BZ4w9C2BP2Z9cRjSkTGz8n4vwmQWH+o7zxiIOpfmxOVNasenGi3H8zHG+FXHjQyQUsRIeIui68lDJFNzSZo5v9E3aJA1yvzOWP3iTGHJTZsQJxVBfzJ8YB4KjPmEqXf6a6hJ49o1OnPjtJ/OGQVVB7CzEjk4sCw5XQpE6sLqkgaRzrqrr8Bi4//7ykF824OLzZ45vDNx3MGsJwawsEKnL9aRZ4wPHwp4jpkneJ1Q07vn0ZtE1tCFl4r/fdbTyWjHPXHcu1DPI80JxOg0i1IWkb09vxld3uRshB68P/L/YyYr5b0iZMA0KGHBe1mIMfm5IVSOD1oYk0llb2lF1tl4MfgrVoQnkq7t/+OSpeO8JM9yRqeckIHRUSdNXH/lIKWuxgD9/sd5h+VDKXa8iohWOxDIu7CQiupKI2omovbMzXC4oFSrGxstQDEak0vjOnzsFxx4wFlctPBSnHjxBLb0I100RPB3kxm4a9uKBH146D7d9ZL7LGHjlEhlEk2KJ9ucWHho4BgQZccI0lAtoeEVTxSepSxpKBiM2Rj4yKCQstdioRMNih2r1l2VdDAYetqgD8Dcg06BAJyhPXK+/4SJcc+EcAPbcxaYbL8bR08e4v2fESUzHt5yX68zxjfjbVxYAAO684iR8zNn8WAW/tON92S8bcKn+HTdzbOBeYxuTuP0j8wXXUD87BrxNiAH/++NlGMXAo9LPn2GaFAiSxe/Zk874Og2DJAbulJ+4wlmFVMJAfcIIyEw8+qNIYHh9IcWoa2yjzdQ7e9JKGcTbhzZ8b1vT8L+38U0pjG9KCc918qpg4N49vPYdnPvIP4orBsUa8FsBHAJgHoDtAG4KO5ExdhtjbD5jbH5bW1uRjysOfPg6qcUzuCrPlDGNSTzwmdNw0MQmTGiuw/obgpK+OGEoxhQPSij2/0XzpmNic52neQ/mkDQNHwNWaaCzJ7fgPy85MnA8k7P8DdUgpUzCDZxqEiVlqiUUubGHSQZhumjSpICGyu8jG6xQCcVhY4DX4O658hT87LITfOeNafSG1QZRwNMkat7D9SUXzrF1b09CM4WRzdjGJA5wWHt90sS4kNEZ4O/wxLoiGwy5Id/98VPw5BfP9B07bHILzjlycmBxlkhSxHfRkDLxp0+diheuXegeK5SBkyuhGG46w+SpgYyFxjr/qFYsU15+vK6LdYIJ4yu+Clpm4AuPmIQfXjoPVwkjarG+yPMjfN6qszutJGj8iSYF38fZcybhncdOw6fOOtRnG1KmXxPndURm4OI7SZreCDtq7qOcKMqAM8Z2MsZyjDELwM8BnFTeZJUH+/ptDw/uugcU3xOKL0AMUyozwIBk4LzQ7oGMTxOXr+WLRVSsErAD5TdJjUbVSBPCBJIMw1DrmvKxMLe5sKBVCSGGjHgvVQwP3phln9j6hOcHzo3hKQdPwFRntMOXnZ8tTEKZBrkBiTgMRTq8+DdOeiVXRDGJDN5qwXzbiXE01yV8E25Rmrmss9YnTXdRFwD85yVH4jv/coyTTk9ukyEa1/qkiRMOHI+pYxq83ws04K73kul1GvI9xBXGYsA2sQyJvLS5DFwYLeQsj81yAy77yo9vTGHRvOlYNG+ae0w0vHK6fAZcQdC4nGGawR3tiQj/+8HjML4p5esYUkKIDcCrV0EG7tf+VQuVgOJXSOdDUQaciKYKX98NYGXYudUE39zWx8CL7AnFhlcnvCiZ6co+p/y8roEsmuuTPqMrGuSxTUn3etVE3NZ9/X4tP2QSk4fpnBQSw7o+YeJth0zAjz90nHtMZjRhBjzMq6Mu4UXLEw2LoZBQXAaeCTJw0Q+cQzZeE5rrXNdNkyiw2tI0gPs+/TYs+fez3GOiUeLXcXzznXN9jbInnXMNj5x20YiI4V1XfvN8H8sMi9chpkGEaJDeNW+aO7nJz1VNKIvyhqq+FBreQPYDTxhBuU2s60nTcA2x2FEnBe+oeoUGnrUsd4SSStgjQm7svnHJkVj9rQvc93HopBb86VN26Gexw5fTxeetdvWklaSGl17CoAADFw2tvPemSKTcvEojDbE/HtuYEiSU/GtSyoG8fi1EdDeAswBMJKItAK4DcBYRzYNdNpsAfKIiqSsRfAJJlDyKLcik1PDqnRWWAQ1Warxiw26pT4Qa8HGNKWze04+sZYWyJ3/4WEOpcy91tls75eAJ7g5BIgyDcNfHTwFgM52lm/dhjbBrCk+nCmGTrq0NSaVOGzWJKe+JWJ8UGbhnsPj8xZmO15CdviS6BrIwFAzcNMj1BuHgBoMPd3marlpwKCY21/kWXXT3Z1DvTFLLqwt54/+Pi4/Ax844GL/5x5uq4ohkWyrjLrNpjkSEARdHEar3UqqEYhoU0K59UTZNm33mnLg1KeE6nn+eBlFm4J4+yDmSXtLE8+ttd8ZkwggQIt5GIhl4k1cHVLIir06myoALhlbMbyph+ILSGQoJRWbg45qS6OjmDNw/wqwUA89rwBljH1Qc/kUF0lJ28Jflk1DKwMAB72XLjefCo6b6vouVraUu4fsuDkm5IeofzEUOf4m8/T5Vk5jjmpLYuq8fJxw4zn1+2MTN2w6diLcdOhFfune573g4A1enq7U+KbihiZ1MUAPnFfnul97yHW9Iehq4yLqnjW3AC9cu9I2iml25CaESigiXgUsG3PUsELLVNZAJZ+DOBtT5jGNYeFfx2WHnq3aEVxlwsY7Uq+Y7ivZC8f5HMfCE6cV7qU+a7oggYZKbH94ZiR4ZdvhbAjL2KEHs0FTyZpPTafg0cImEjW3wJE0lA2dejBQuoVx09BQ88uqOcAae8DNw/pOv3CUNfGxDyn0+l4UObmvChs7eyFFZKagMrx8mOGKqHdBf9CcNi7GRD/IL4I1frNSv/Oe5+LS0UKYuwMD9ga44xjo6Xn8mF2pw01kvnrLthRJsuL+4/ET84ZOnug1y+tiGwDkyAhp4CAMPk1BaGxJujBnRgNsrCP33PqRN7fucShjutbK5mjqmwde4uPeOQWoGLoMbFFGnBTyDIRr9/f0Zd2QT3I3GctMaBX7fI6e24oZ3H40PneytT8hX/0Rjzjse1SIXsZzVDLwwCYXfTmTgshuhaj0FPy5OfiYkBi5uViIG37Lj2HhMVRU1kD9TdDtVzTPxOSRVHbUUDPzL59ueSWce5jlWyJ2Jz4ALE8rvONbW5lvqE775k7GNSbfc+waz9v2EZ1cCIyIaYRju/vjJ2NE14BvOmMUycKnhJRQGXOVD7mPg0qo08Z2Ob/QWNXD2PG1MPbYJm0ikMznUJ00MZCyft4SIya31mNxaj950FhOb6/CtRXOxp3fQ3cBVBZm1hDLwEKPgY+ASg5SL+6jpY/Chk2firhf9DDxpemw9yo0Q8IbVphFcoahi4EmJcYvpk6/J5JhbrgEJhQdAy2McuXFLJgzXePP8FrLlHU+XHJYUkLxQFPWgaAYuzBeIk4+AFGVTMuA+eU9i4PJmJbztpBJG6K5A3r3tdz1JlEEVI5yxTUl0p7MhGjhn4IZvVP7i1872hc4Q2zgR+dZSiEz7fy+dh69dNAct9Um33BqcrRk9V0vb64wp7l1OjGgGPrYxhTlTWv2LLIrsCQMM3HlxjSGslENsSM11Cdu1TmFQTnAWLxw4vhFzp43Bym+ej3+T/I4HBM1ddFlSoakugfb/OAdnzG7DonnT8bEzDo6VRiBcAw/zQmmtT7p5aUr5h52mFPtjzpRWZQdhM7dwrwsR/HomxWEGohm4zPDCYqhzCUXexPqQNlsbnzEuelTD3688b1Io+H2UDDzvJGb+pj1nSgvOOcKOahhwIzSDboS27u1PG2AbLy+SpPebanOMjLBtXUryAVe9u4aUif/94HH4zRWeo5tKjuBSiyrfIgM/6/A25zwTk1vrfZ1BsB4EJRTALivu8cOPj3MImOiFkjDIrUPDygul1iA23OINuMTAzSADV8HPwBO+Y2K63nnsNDzyuTNw4dG2ht5cl8DbD/P7zXMGDiCUgReDQPjUEAZ+imKTYMCWXHi5jhX8tMXATsfNHIt1374IR88Y4zY2EQlTHYtC+TwnfX2DWcydNsZ5lv1b1CIsmZ3z8pdJO2f48qjlMwsOxZ8+dSrmx1yVGXcz5DBc5NQFVbmL+2Cq6gE3ZAdPbMInzjzYNfLXXjgHRzrS4r8cP8ONzyFrvKbCC0UM9OWXULzdbCzGBAklyMD5alfAHtH1DEYbcMBuGz4XSUW58vwp24SwEvN/P3gcnvvqAuWzDhjXiKOnj8EtH7Djnfg1cHXaeF7GOBKo6AeeSngMXBvwEiAWXrEFGZiMcxfMRKtQYmXj2nKd20j89zxS2oR39uQWfO7s2e73AUEDT5rhG90WiqAfuF9XntCUwvPXLHRXyMkQY2HIwcDEJcj8s0pjFyWUfGhyg5flcNDEJmy44SJceNQU+3lKCcXwpUVMt/ifg7NSORyraRBOONAz3l+9YI5P3xbzAiCwkrFQnHzwBGy68WJ3LgcIBp0C1CSC15OGlIlrLzrC9eZ5++FtLjOuSxquf73sOWJSsAMyhDADYgfSkPKkAzBxLUKQgWctbz/UxpTpG23F7cBV+W2MZODcD9xO/4xxwZAKgL1I7C+fPR3vPm5G4Dlhu/rw+tYqkbOedBYJw3DzVxnzPcI1cA6xgUZtjRQFWcOKivonQjSOfNUgPxZHDxWH4TmL+Rh4uRBnEnPa2IZIHZ0Xj2zAVXKRapf3hGHEZqycgfPhtxiPWnULj4H7j6s08Ac+cxoOGN+I7733GEzLMwGsiuxo39eZpKuA50HCIGRyLO8kpvxOxRg8CTd9ni83bxduJE3DcMMA1CcNDGTsuOPiJDqHOIlp7x9p+O519AwhdEHWcg14YA1FzDqtYtmcgavmJ0Q/8EKg8gOXwW8p+7z3prMY05B09fcizU5ejBIGXtx1c5ywsYDHquSher4NfUUGLksoceqTPOnqxRov36sLSij+RhBnGMiZnGw4RO8EDlUIgYQZvm2YjEMm2Z4s4qSxyk/XS4OaacsGPGEQjj1gLADgffMPwGmHTkQx4J1uJSau3AVTIjFQhCkV424DnoFJZy23s00lDG+jDOLXcXIBd33APKdMRAYujpYaU6Zbhxi8/PO6ev7cKfjLVacDsP36efS+xpSJhz57unuf2Aw8woCr2qNVpA4dR0IxpJEL90O3mN2uPAZeGQs+Shh44Q3pxa+d7dOCPVc0vzHIp0OLDY0PY92l2jG6ZblSuwy8jOwuyNbUeVKll8sqYRq061PtY+CqScz4ktC75k3H2IaUb45AFWxIvDeg0MAlCaVQz40wuBp4me4nws6Lfx/MCU3BVbc8L9wHm7tbZp1t4wD7PctRHvlEtWUBA46L3zEzxuIfG/aABAYux2Jxnyfs4C7Wo6NnjMEPL52HUw+ZgNO/87R9XdLEUdPH4KCJTdi4qzc2A49ym1QycFcDL+x9+BbyhFzKy4/nf1xj0lurYRL4HG2lGPioMOC8XogxJ/JBXL0JiD7D9nc3dkSeSudbuOPodLzyGwbZk3sRu/HIlc71QqmghBJmSFWV+PvvO9Z/Dqk7HFHr5AacV3TArux8GP/l8w+PTC8RYcGc4M5CgLqTkScxxZV59nH7e7kMuOtGWEAne+8nTo3VgalCFqjSXScYVAC46f3H4ld/34TjZ45zy0O8jtzr7PeVtSzc9fFTsHTzXux3QlIw5l0j1kv/JKaXf9lradG86QAQkFASMdsSB7/urMPbcP075gLwCE3UiLgiDFySnhpTCcyaYHdISdPAde84HN94YBUmtca3PYVgVBhwPow8cda4ou/BK6g86ZPPZ5k3tHGN4nJfzwvl/k+fFnm9OFRdOGdSQAN/7PNnKsPSFgJ54keu6HyCJg6DkeUq3gjEIEJcQmmpS6DL8QPmxm7TjRcXkHIP/DWoWJwczMpNK/kNe6HLz8PgTjQXwPjE7biiIIeYDU1D0m/AJ7XU46sX2ItXTLcjY16nJqU9ZzEcPWMMjp4xBj9Zss69lxz0CvC7EcLnRqiul/xdcULjvp+YNJXf96SDxmMWjxujWAkso1ANXLxX6AYjXHoSOo45U1qwcVcvEqaB8+ZOwXlzpxT03EIwKgz4QROb8NMPH+9bdVUoAhq4u8gi+rqEaeAn/3q8Lz6HGDciH/g5h09uwc8uOwHfeGCle18AOFzQ6YuFrIGbBuG/Fs3FgROasLajB+cdaXtlyPboG4qwt0EGbl8k7oTCG2CzYMDLpReryjQpyV8c3AhxL4lyMXC+2nR3b/ikb7GQA3OFge/eruIXIvngRcINPTfgou85N3yMefVerDNi5ELLiY0ChC/84uC6NZea4jJkTgoGBoOrM6PoVCkT/2GXckIndv6HTW7BX1fuKCyofrHpqvgThgkuOGqqLxhPoZCNwKcX2B4Iqp23ZVx09FTlJhBxPGI4i2uuT/jiiZczQLzMWkwiXHbqLJx5WBuuOP0gNya2yJBufv+xvoVGfEm0vAiIT+qI4QF4pMSvOhstAOWLl6yUUBSeMID3LuOusIyLw6fYbn/rFTuTlwp3x5w8LVeWUHz3cMrankz0y0r8fYmuf3wOicHbzEQOYSBOYiYMOz58PgmJSyFcDozrIcZXSIrbErq7akVsBVhKHQvd2V5Rd7j82jWQVV5TTowKBl4OJCQGvnDOZN9w/wPzD4h9L8484njHyDqtygugVMjkN2yCVGQwstHnLn2t9Un85arTsWVvHwCPLWUkCYWX3dX3LANQOgPnzTZSQpEZuHMud318xzHTUA4c7uyZKu7G84m3H4yOrtIZuew5o9rRBxAmMRX2jG+WrJKMohi4xRi+995jccXpB2HutDH4yp9WuOeIboQnHDgOb+3py2uQXQbuXCtv3B0GXqf6ffFR7P+5CNZbCukJu5QTE3H0xvcLkHdjqgS0AY+JME8GoHDdllfAWF4ovONwmNKkljo01yXK6qImuziFpUtskLK+yRn4mIakq52K5+Xb5b58DDz83nIj5Ma+raUOy79xXmgIgUIxpjGJL59/OM6Y7bkhXnvhEWW5d0KQCp776oLQPVxFLVvGVy+cg4PbmnHOEZOx5I1O935h14lRIhtSJo5z5MC/fWUB9vbZm6ZwA8aYPdq9QIrKqUJj0plbMf0joXzgdap/0Dufv8uozbjDWHQchHVGPEqi2BlOdEaY+5yyqSS0AY8JWQMvBXyYGsdtymW9zr8PnTwT5wrbbVUC+SZmgfAJP3mDBa6Bh+1mw1GqX3tUzImwlZgiIxO3aisH5A22ywUxxGzYikJANKjBd9mYSuDyt80CIKwQdDVwzwuFw9PA/fc6YHyjK68VEzaASyhcfsnk6eTl6waETR7C0iiiGNKTMMjdK1UFvtGEz4A7bp0RfUnZkDdHzqbFHUS0Ujg2noieIKK1zv/i3TtqDKX04hzuYolYfuDc+8VGfdJ0G02lkI8t83SoIGujKglFhXIxcKUfuBQPnKOSnWClwNOc7x3JC3nC4GrHrgbO5YwgA4+6V9h+qVHgpEiuM/nAJ0fFScwrzzgE75o3DZedOiv0umLWTogLm1RwGbjQHia2hO+dWm7EKfVfA7hAOnYNgMWMsdkAFjvfRzRa65N493HT8cv/d2LJ96pTLG4Jg7chb8mPDYd0bznCnwphDFxmsvUxDXipAe+9jWuD90klDDTXJTBWkhvK0RkPNXhgq3wjBs7AozRhIFiv+HsV2XCcML/FePBwWeK6d8zF586ejYUhvv0yuO7/EWcUAdjl8YNLj4vsDIrRwOvyOBxwDVyMnV6Ks0ShiLMjz7NENEs6vAj2NmsAcAeAJQC+Ws6EDTcYBuGWD8wry704O8pn1ACPNVRqKS4AnHzQeJw4axz29WWwtqMn1lA2LLRss1R5uYQStkkFR7k0/bBJzMe/cGZgIVcN2m98/eIj8OFTZubdqKMuQkJRgUkSihiDXPRCCUMhHfCsCY3YtLvP/T6mMYkvnntY7OsnNNcVtV6gmBFXvtGyx8D99bcxZeLio/PPA5SKYruKyYyx7QDAGNtORPG6Tg0AXqXIZ9QA0QWuculpTCXwh0++DVffsxRrO3piMfAwH1/ZgHoMPNqQlEvOCNMq8wWmqhUkTQOHTsrv+18X4YUigpeWPIkpeqGIC39C01UAA3/ws6f7NnIYKhTFwPPELeI7Bckj0te+JYsWlUHF/cCJ6Eoiaiei9s7Ozko/ribAX3a+iT1AWGE4BHSR7wQUNTnGEcbAZdQrVmKqUGrs7EpvXVVrSAhufVEg1zjb3+sUIWB5kUb164Vo4K31yap0qMUxcG/7PhU8L5TyrCEoFMUy8J1ENNVh31MBdISdyBi7DcBtADB//vwhmJcd/vAMeC7PmdHD1nLjslMOxKkHT8DsyfkZnszAf/3RE5XhZuNuOlHJSczRCC5dnZRn8wkOj4FzLxSv5smrNVUouQMeAhQj0/EOLYw/LZo3Dc+80VmWFdHFoFgD/iCAywHc6Px/oGwpGgXgjYRvkhsFPmwtNo55ISCiWMYbCDLwsw5Xq2j1MYfWJbsROiZI228bjakEHvv8mZiZx2OJVysmLaXP+Qy453seBs5uw3ZzGg4oRQMPm4N6z/Ez8O7jpg9J+1Qhb2kT0d2wJywnEtEWANfBNtz3EtEVAN4C8L5KJnKkgc/Yx1m4UOkdPYpF3CEjH8rna9jlYuCVnOytNcRhhXJ5cYN12SkHuseMGBo4ANzygWNx3AHD16O4GA085XrlhLfVahlvIJ4XygdDfjq7zGkZNXAllDgMvMI7ehSLQtjM/33oeBw1vTXynFK16yGIGzQi4TFw/p3wxn9f6PMq4Z/yTYjyrciGK4qR19zRcoz5qmpg+I53RjDqkgVo4MOUgReCi4/J705VCxrqSISqXgU3M7b/x3VJHEkoxGOsGtCtpgoY6wRPmhBjg4nDHE2aB8MfqSiX98hwG6kMd7gMPELh5jLLUCwNH24oxOGgGtAGvAo4fuY4/OAD83D9O+fmPfeA8Y3YcMNFeNdxI9uAl4pCyOEVZxwEADhyarSsMxpwobPY5IzZ4bHy43ihDGd8a9Fc3/62hYCHhk0M0xGillCqhEIMsnaNi484JbXg8ElF7/wz0nD8zHF5y6Kak3TlwEdOnYWPRMRIicIXzj0MU8bU45IhWFVZDLQB1xgRiJIANEqDIU10jibUJ0189LSDqp2MUAzPcYGGRpGocbI4LOFFLByFFnyYQxtwDQ2NSHhuhNqADzdoCUWjIHz5/MOLiv1caWjbUjnwkKqt9eXd9EKjdGgDrlEQyr3TzLjGJPb2lXPvQK2hlBvHzxyL/7zkSLxHe0INO2gDrlFVLPnyAvQPlu5jqwl45UBEuOL04TuRN5qhDbhGVTGmIVnwllpR0JOYGqMJw0/M1NAoAloD1xiN0AZcY0Tg3CPtcLaHTmqucko0NIYOWkLRGBF4//wDcPEx04Z1PGoNjXJDM3CNEQEi0sZbY9RBG3ANDQ2NGkVJlIWINgHoBpADkGWMzS9HojQ0NDQ08qMcY84FjLFdZbiPhoaGhkYB0BKKhoaGRo2iVAbOADxORAzAzxhjt8knENGVAK50vvYQ0etFPmsigNHG9HWeRwd0nkcHSsnzgaqDVMo+d0Q0jTG2jYgmAXgCwGcZY88WfcPoZ7WPNo1d53l0QOd5dKASeS5JQmGMbXP+dwC4H8BJ5UiUhoaGhkZ+FG3AiaiJiFr4ZwDnAVhZroRpaGhoaESjFA18MoD7nf3yEgDuYow9WpZUqRHQ10cBdJ5HB3SeRwfKnueSNHANDQ0NjepBuxFqaGho1Ci0AdfQ0NCoUdSEASeiC4jodSJaR0TXVDs95QIR/ZKIOohopXBsPBE9QURrnf/jhN+udcrgdSI6vzqpLh5EdAARPU1Eq4loFRFd7RwfyXmuJ6KXiGi5k+dvOsdHbJ45iMgkoqVE9JDzfUTnmYg2EdGrRLSMiNqdY5XNM2NsWP8BMAGsB3AwgBSA5QCOrHa6ypS3MwEcD2ClcOy7AK5xPl8D4DvO5yOdvNcBOMgpE7PaeSgwv1MBHO98bgHwhpOvkZxnAtDsfE4CeBHAKSM5z0LevwjgLgAPOd9HdJ4BbAIwUTpW0TzXAgM/CcA6xtgGxtgggHsALKpymsoCZi962iMdXgTgDufzHQDeJRy/hzGWZoxtBLAONeZ3zxjbzhh7xfncDWA1gOkY2XlmjLEe52vS+WMYwXkGACKaAeBiALcLh0d0nkNQ0TzXggGfDmCz8H2Lc2ykYjJjbDtgGzwAk5zjI6ociGgWgONgM9IRnWdHSlgGoAPAE4yxEZ9nAD8A8BUAlnBspOeZhxZ52QkhAlQ4z7UQAV+1Te1o9H0cMeVARM0A/gTg84yxLgrfiXhE5JkxlgMwj4jGwl47cVTE6TWfZyK6BEAHY+xlIjorziWKYzWVZwenMSG0CBGtiTi3LHmuBQa+BcABwvcZALZVKS1DgZ1ENBUAnP8dzvERUQ5ElIRtvH/HGLvPOTyi88zBGNsHYAmACzCy83wagHc6+wXcA2AhEf0WIzvPYOrQIhXNcy0Y8H8CmE1EBxFRCsClAB6scpoqiQcBXO58vhzAA8LxS4mojogOAjAbwEtVSF/RIJtq/wLAasbYzcJPIznPbQ7zBhE1ADgHwBqM4Dwzxq5ljM1gjM2C3V6fYox9GCM4zxGhRSqb52rP3Mac3b0ItsfCegBfr3Z6ypivuwFsB5CB3SNfAWACgMUA1jr/xwvnf90pg9cBXFjt9BeR39NhDxNXAFjm/F00wvN8DIClTp5XAviGc3zE5lnK/1nwvFBGbJ5he8ktd/5WcTtV6TzrpfQaGhoaNYpakFA0NDQ0NBTQBlxDQ0OjRqENuIaGhkaNQhtwDQ0NjRqFNuAaGhoaNQptwDU0NDRqFNqAa2hoaNQo/j+gr+5N3uFe7AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses)\n",
    "plt.ylim(5,50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用autograd实现的线性回归最大的不同点就在于autograd不需要计算反向传播，可以自动计算微分。这点不单是在深度学习，在许多机器学习的问题中都很有用。另外需要注意的是在每次反向传播之前要记得先把梯度清零。\n",
    "\n",
    "本章主要介绍了PyTorch中两个基础底层的数据结构：Tensor和autograd中的Variable。Tensor是一个类似Numpy数组的高效多维数值运算数据结构，有着和Numpy相类似的接口，并提供简单易用的GPU加速。Variable是autograd封装了Tensor并提供自动求导技术的，具有和Tensor几乎一样的接口。`autograd`是PyTorch的自动微分引擎，采用动态计算图技术，能够快速高效的计算导数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:.conda-pytorch] *",
   "language": "python",
   "name": "conda-env-.conda-pytorch-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}