concepts

• linear algebra is a form of continuous rather than discrete mathematics
• detailed information --> Matrix Cookbook (Petersen and Pedersen, 2006)
• Scalars:–>(ponit)
  • A scalar is just a single number
  • $a=a^T$a=aT
• Vectors:–>(line)
  • A vector is an array of numbers, the numbers are arranged in order.A standard column vevtor
• Matrices:–>(face)
  • A matrix is a 2-D array of numbers, so each element is indentified by tow indices.
  • Note:transpose for matrix:$(\mathbf{A}^T)_{i,j}=A_{j,i}$(AT)i,j​=Aj,i​
  • $\mathbf{C} = \mathbf{A}+\mathbf{B}$C=A+B where $C_{i,j}=A_{i,j}+B_{i,j}$Ci,j​=Ai,j​+Bi,j​.
• Tensors:–>(volume)
  • A tensors is an array of numbers arranges on a regular grid with a variable number of axes.

NOTE:
$\mathbf{D}=a\cdot \mathbf{B}+c,\text{ where }D_{i,j} = a\cdot B_{i,j}+c$D=a⋅B+c, where Di,j​=a⋅Bi,j​+c
Broadcasting:
$\mathbf{C} = \mathbf{A}+\mathbf{b},\text{ where }C_{i,j} = A_{i,j}+b_{j}.$C=A+b, where Ci,j​=Ai,j​+bj​.

• matrix product:
  • $\mathbf{C} = \mathbf{AB}$C=AB, where $C_{i,j}=\sum\limits_k A_{i,k}B_{k.j}$Ci,j​=k∑​Ai,k​Bk.j​
  • also called element-wise product or Hadamard product and denoted as $\mathbf{A}\odot \mathbf{B}$A⊙B
  • distribution:$\mathbf{A(B+C)=AB+AC}$A(B+C)=AB+AC
  • association:$\mathbf{A(BC)=(AB)C}$A(BC)=(AB)C
  • uncommutation: $\mathbf{AB\neq BA}$AB̸​=BA
  • transpose: $(\mathbf{AB})^T=\mathbf{B}^T\mathbf{A}^T$(AB)T=BTAT
• dot product: (for two vector $\mathbf{x}$x and $\mathbf{y}$y)
  • likely martix product $\mathbf{x^Ty}$xTy
  • communication:$\mathbf{x^Ty=y^Tx}$xTy=yTx
• linear equations:
  • $\mathbf{Ax=b}$Ax=b, where $\mathbf{A}\in\mathbb{R}^{m\times n}$A∈Rm×n is a known matrix, $\mathbf{b}\in\mathbb{R}^m$b∈Rm is a known vector, and $\mathbf{x}\in\mathbb{R}^n$x∈Rn is a vector of unknown variables to be solved.
• matrix inversion
  • denoted by $\mathbf{A}^{-1}$A−1, and define as $\mathbf{A}^{-1}\mathbf{A=I}_n$A−1A=In​
• identity matrix
  • $\mathbf{I}_n\in\mathbb{R}^{n\times n}$In​∈Rn×n, and we have $\forall \mathbf{x}\in\mathbb{R}^n,\mathbf{I}_n\mathbf{x=x}$∀x∈Rn,In​x=x

Linear Dependence

• The geometry meaning of linear equations
  $\mathbf{A}=\begin{bmatrix} a_{1,1}&a_{1,2}&\cdots&a_{1,N}\\ a_{2,1}&a_{2,2}&\cdots&a_{2,N}\\ \vdots&\vdots&\ddots&\vdots\\ a_{N,1}&a_{N,2}&\cdots&a_{N,N} \end{bmatrix} = \begin{bmatrix} \mathbf{A}_{:,1}&\mathbf{A}_{:,2}&\cdots&\mathbf{A}_{:,N} \end{bmatrix}$A=⎣⎢⎢⎢⎡​a1,1​a2,1​⋮aN,1​​a1,2​a2,2​⋮aN,2​​⋯⋯⋱⋯​a1,N​a2,N​⋮aN,N​​⎦⎥⎥⎥⎤​=[A:,1​​A:,2​​⋯​A:,N​​]
  
• linear combination
  • $\{v^{(1)},\cdots,v^{(n)}\}${v(1),⋯,v(n)} is given by $\sum\limits_ic_iv^{(i)}$i∑​ci​v(i)
• Span
  • the span of a set of vectors is the set of all points obtainable by linear combination of the original vectors.
  • if $\mathbf{Ax=b}$Ax=b has a solution, $\mathbf{b}$b is in the span of the columns of $\mathbf{A}$A.–> column space or the range of $\mathbf{A}$A.
  • $n\geq m$n≥m is the necessary condition
  • A set of vectors is linearly independent if no vector om the set is a linear combination of the oter vectors.
  • A square matrix with linearly dependent colums is known as singular.
  • For square matrices, the left inverse and the right inverse are equal.
• Norms
  • measure the size of vectors
  • $L^p$Lp norm is given by $\|\mathbf{x}\|_p=(\sum\limits_i|x_i|^p)^{\frac{1}{p}}$∥x∥p​=(i∑​∣xi​∣p)p1​ for $p\in\mathbb{R},p\geq1$p∈R,p≥1
  • functions mapping vectors to non-negative values.
  • satisfies the properties:
    • $f(\mathbf{x})=0$f(x)=0=>$\mathbf{x}=0$x=0
    • $f(\mathbf{x+y})\leq f(\mathbf{x})+f(\mathbf{y})$f(x+y)≤f(x)+f(y) (the triangle inequality)
    • $\forall\alpha\in\mathbb{R},f(\alpha\mathbf{x})=|\alpha f(\mathbf{x})|$∀α∈R,f(αx)=∣αf(x)∣
  • $L^2$L2 is the Euclidean norm between the origin and the point $\mathbf{x}$x.
    • denoted as $\|\mathbf{x}\|$∥x∥, calculated as $\mathbf{x}^2$x2.
    • increases very slowly near the origin.
  • $L^1$L1 is the grows at the same rate in all locations.
    • $\|\mathbf{x}\|_1=\sum\limits_i|x_i|$∥x∥1​=i∑​∣xi​∣
  • $L^0$L0 measure the size of the vector by counting its number of nonzero elements.
    • not a norm, always used $L^1$L1 as a substitute.
  • $L^{\infty}$L∞ known as the max norm
    • $\|\mathbf{x}\|_{\infty}=\max\limits_i|x_i|$∥x∥∞​=imax​∣xi​∣
  • $L^2$L2 of matrix is Frobenius norm
    • used to measure the size of a matrix
    • $\|A\|_F=\sqrt{\sum\limits_{i,j}A^2_{i,j}}$∥A∥F​=i,j∑​Ai,j2​​
  • Represented dot product by norms: $\mathbf{x}^T\mathbf{y}=\|\mathbf{x}\|_2\|\mathbf{y}\|_2\cos\theta$xTy=∥x∥2​∥y∥2​cosθ, where $\theta$θ is the angle between $\mathbf{x}$x and $\mathbf{y}$y.
• Special Kinds of Matrices and Vectors
  • Diagonal matrices:
    • $\mathbf{D}$D is a diagonal, if and only if $D_{i,j}=0$Di,j​=0 for all $i \neq j$i̸​=j.
    • diag($\mathbf{v}$v) denote the diagonal matrix whose diagonal entries are given by the vector $\mathbf{v}$v.
    • diag($\mathbf{v}$v)$\mathbf{x}=\mathbf{v}\odot\mathbf{x}$x=v⊙x
    • if every diagonal entry is nonzero, diag($\mathbf{v}$v)$^{-1}$−1=diag([$1/v_1,\cdots,1/v_n$1/v1​,⋯,1/vn​]).
    • for a non-square diagonal matrix $\mathbf{D}$D, if $\mathbf{D}$D is taller than it is wide, concatenating some zeros to the results; if $\mathbf{D}$D is wider than it is tall, discarding some of the last elements of the vector.
  • symmetric matrix:
    • $\mathbf{A}=\mathbf{A}^T$A=AT
  • unit vector
    • unit norm: $\|\mathbf{x}\|_2=1$∥x∥2​=1
    • orthogonal: $\mathbf{x}^T\mathbf{y}=0$xTy=0
    • orthonormal: orthogonal and unit
  • orthogonal matrix:
    • $\mathbf{A}^T\mathbf{A}=\mathbf{A}\mathbf{A}^T=\mathbf{I}$ATA=AAT=I
    • Implies: $\mathbf{A}^{-1}=\mathbf{A}^T$A−1=AT
• Eigendecomposition
  • decompose a matrix into a set of eigenvectors and eigenvalues.
  • eigenvector and eigenvalues
    • a square matrix $\mathbf{A}$A, a non-zero vector $\mathbf{v}$v,satisfied $\mathbf{Av}=\lambda\mathbf{v}$Av=λv.
    • $\lambda$λ is the eigenvalue corresponding to this eigenvector $\mathbf{v}$v. ++Also there are left eigenvector $\mathbf{v}^T\mathbf{A}=\lambda\mathbf{v}^T$vTA=λvT++
    • for $s\in\mathbb{R},s\neq0$s∈R,s̸​=0, if $\mathbf{v}$v is an eigenvector of $\mathbf{A}$A, then $s\mathbf{v}$sv has the same eigenvalue.
    • let $\mathbf{V}=[\mathbf{v}^{(1)},\cdots,\mathbf{v}^{(1)}],\mathbf{\lambda}=[\lambda_1,\cdots,\lambda_n]^T$V=[v(1),⋯,v(1)],λ=[λ1​,⋯,λn​]T, then $\mathbf{V}diag(\mathbf{\lambda})=\mathbf{AV}$Vdiag(λ)=AV, namely $\mathbf{A}=\mathbf{V}diag(\mathbf{\lambda})\mathbf{V}^{-1}$A=Vdiag(λ)V−1.
      
  • every real symmetric matrix can be decomposed
    • $\mathbf{A=Q\Lambda Q}^T$A=QΛQT, where $\mathbf{Q}$Q an orthogonal matrix composed of eigenvectors of $\mathbf{A}$A and $\mathbf{\Lambda}$Λ is a diagonal matrix.
    • the eigenvalue $\Lambda_{i,i}$Λi,i​ is associated with the eigenvector $Q_{:,i}$Q:,i​
    • is not unique.
    • if any of eigenvalues are zero, the matrix is singluar
    • eigendecomposition:
      • $f(\mathbf{x})=\mathbf{x}^T\mathbf{Ax}$f(x)=xTAx, subject to $\|\mathbf{x}\|_2=1$∥x∥2​=1.
      • if $\mathbf{x}$x is the eigenvector of $\mathbf{A}$A, $f$f is the eigenvalue.
      • the maximum/minimum value of $f$f is the maximum/minimum eigenvalue
      • positive definite: a matrix with all positive eigenvalue.$\mathbf{x}^T\mathbf{Ax}=0$xTAx=0==>$\mathbf{x}=0$x=0
      • positive semidefinite: a matrix with all positive or zero eigenvalue.==>$\forall \mathbf{x},\mathbf{x}^T\mathbf{Ax}\geq 0$∀x,xTAx≥0
      • negative definite
      • negative semidefinite
• Singular Value Decompsition:
  • every real matrix has a singular value decomposition
  • $\mathbf{A=UDV}^T,\mathbf{A}_{m\times n},\mathbf{U}_{m\times m},\mathbf{D}_{m\times n},\mathbf{V}_{n\times n}$A=UDVT,Am×n​,Um×m​,Dm×n​,Vn×n​.
  • $\mathbf{U,V}$U,V is the orthogonal matrix
  • $\mathbf{D}$D is the diagonal matrix and is the singular value of $\mathbf{A}$A
  • the columns of $\mathbf{U}$U are the left-singular vectors.
  • the columns of $\mathbf{V}$V are the right-singular vectors.
• The Moore-Penrose Pseudoinverse
  • define: $\mathbf{A}^+=\lim\limits_{\alpha\searrow 0}(\mathbf{A}^T\mathbf{A}+\alpha\mathbf{I})^{-1}\mathbf{A}^T$A+=α↘0lim​(ATA+αI)−1AT
  • $\mathbf{AA}^+\mathbf{A=A}$AA+A=A
  • $\mathbf{A}^+\mathbf{AA}^+=\mathbf{A}^+$A+AA+=A+
  • $\mathbf{AA}^+,\mathbf{A}^+\mathbf{A}$AA+,A+A are symmetric
  • Compute: $\mathbf{A}^+=\mathbf{VD}^+\mathbf{U}^T$A+=VD+UT, where $\mathbf{U,V,D}$U,V,D are the singular value decomposition of $\mathbf{A}$A.
  • if $\mathbf{A}$A is a wide matrix, $\mathbf{x=A}^+\mathbf{y}$x=A+y with the minimal Euclidean norm among all possible solutions.
  • if $\mathbf{A}$A is a tall matrix, $\mathbf{Ax}$Ax is as close as possible to $\mathbf{y}$y.
• Trace Operator
  • Define: $\text{Tr}(\mathbf{A})=\sum\limits_i\mathbf{A}_{i,i}$Tr(A)=i∑​Ai,i​
  • Frobenius norm of a matrix: $\|A\|_F=\sqrt{\text{Tr}(\mathbf{AA}^T)}$∥A∥F​=Tr(AAT)​
  • $\text{Tr}(\mathbf{A})=\text{Tr}(\mathbf{A}^T)$Tr(A)=Tr(AT)
  • $\text{Tr}(\mathbf{ABC})=\text{Tr}(\mathbf{CAB})=\text{Tr}(\mathbf{BCA})$Tr(ABC)=Tr(CAB)=Tr(BCA), more generally, $\text{Tr}(\prod\limits_{i=1}^n)=\text{Tr}(\mathbf{F}^{(n)}\prod\limits_{i=1}^{n-1}\mathbf{F}^{(i)})$Tr(i=1∏n​)=Tr(F(n)i=1∏n−1​F(i))
  • $a=\text{Tr}(a)$a=Tr(a)
• The Determinant
  • define for a square matrix: $\det(\mathbf{A})=\prod\limits_{i=1}^n\lambda_i$det(A)=i=1∏n​λi​
  • The absolute value of the determinant can be thought of as a measure of how much multiplication by the matrix expands or contracts space.
  • If the determinant is 0, then space is contracted(收缩) completely along at leastone dimension, causing it to lose all of its volume.
  • If the determinant is 1, thenthe transformation preserves(保留) volume.