最小二乘法公式推导

最小二乘法公式推导

引用

CSDN

1.

https://blog.csdn.net/jackiejane/article/details/144341638

最小二乘法（Least Squares Method）是一种用于寻找最佳拟合数据集的线性模型的方法。它通过最小化观测值与模型预测值之间的误差平方和来找到最优参数。以下是针对简单线性回归模型y = w x + b的最小二乘法推导过程。

该方法常用来进行直线的拟合，以下以该应用来说明：

样本数据为：(x_1, y_1), (x_2, y_2), ..., (x_n, y_n)

• 最小化残差平方和（Residual Sum of Squares, RSS）来估计回归系数：
  [
  \mathcal{L}=\sum_{i=1}^n(y_i-wx_i-b)^2
  ]

• 分别对w与b进行一阶求偏导过程如下:
  [
  \frac{\partial\mathcal{L}}{\partial\mathcal{w}}=\sum_{i=1}^n-2x_i(y_i-b-wx_i) \
  =-2\sum_{i=1}^nx_i(y_i-b-wx_i) \
  =-2(\sum_{i=1}^nx_iy_i-\sum_{i=1}^nbx_i-\sum_{i=1}^nwx_i^2) \
  =-2(\sum_{i=1}^nx_iy_i-nb\bar{x}-w\sum_{i=1}^nx_i^2) \
  ]
  [
  \frac{\partial\mathcal{L}}{\partial\mathcal{b}}=\sum_{i=1}^n-2(y_i-b-wx_i) \
  =-2\sum_{i=1}^n-2(y_i-b-wx_i) \
  =-2(\sum_{i=1}^ny_i-\sum_{i=1}^nb-\sum_{i=1}^nwx_i) \
  =-2(n\bar{y}-nb-nw\bar{x}) \
  =-2(\bar{y}-b-w\bar{x})
  ]

• 令偏导函数等于0，解方程组：
  [
  \begin{cases}
  \bar{y}-b-w\bar{x}=0 \
  \sum_{i=1}^nx_iy_i-nb\bar{x}-w\sum_{i=1}^nx_i^2=0 \
  \end{cases} \
  =>b=\bar{y}-w\bar{x} \
  代入第二个方程 \
  =>\sum_{i=1}^nx_iy_i-n(\bar{y}-w\bar{x})\bar{x}-w\sum_{i=1}^nx_i^2=0 \
  =>\sum_{i=1}^nx_iy_i-n\bar{x}\bar{y}+nw\bar{x}^2-w\sum_{i=1}^nx_i^2=0 \
  =>\sum_{i=1}^nx_iy_i-n\bar{x}\bar{y}=w(\sum_{i=1}^nx_i^2-n\bar{x}^2) \
  =>w=\frac{\sum_{i=1}^nx_iy_i-n\bar{x}\bar{y}}{\sum_{i=1}^nx_i^2-n\bar{x}^2} \
  =>w=\frac{\sum_{i=1}^nx_iy_i-n\bar{x}\bar{y}-n\bar{x}\bar{y}+n\bar{x}\bar{y}}{\sum_{i=1}^nx_i^2-2n\bar{x}^2+n\bar{x}^2} \
  =>w=\frac{\sum_{i=1}^nx_iy_i-\sum_{i=1}^nx_i\bar{y}-\sum_{i=1}^n\bar{x}y_i+\sum_{i=1}^n\bar{x}\bar{y}}{\sum_{i=1}^nx_i^2-\sum_{i=1}^n2\bar{x}x_i+\sum_{i=1}^n\bar{x}^2} \
  =>w=\frac{\sum_{i=1}^n(x_iy_i-x_i\bar{y}-\bar{x}y_i+\bar{x}\bar{y})}{\sum_{i=1}^n(x_i^2-2\bar{x}x_i+\bar{x}^2)} \
  =>w=\frac{\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{\sum_{i=1}^n(x_i-\bar{x})^2}
  ]

示例代码如下：

import numpy as np
import matplotlib.pyplot as plt

def calc_line_param(x, y):
    x_avg = np.mean(x)
    y_avg = np.mean(y)
    w = np.sum((x - x_avg)*(y - y_avg)) / np.sum((x - x_avg) ** 2)
    b = y_avg - w * x_avg
    return w, b

if __name__ == '__main__':
    plt.rcParams['font.family'] = ['SimHei']
    x = np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0])
    y = np.array([1.5, 2.6, 3.7, 4.1, 5.1, 6.2, 7.9])
    plt.scatter(x, y, marker='x', c='r', label='样本点')
    w, b = calc_line_param(x, y)
    _y = w * x + b
    plt.plot(x, _y, c='b', label='拟合直线')
    plt.legend()
    plt.show()

运行效果如下：