Quiz 1 Solution (StatComp 2021/22)

Finn Lindgren

Deriving the Hessian

Define the individual terms as \(f_k=y_k\log(1+e^{-\eta_k}) + (N-y_k)\log(1+e^{\eta_k})\), so that \(f=\sum_{k=1}^n f_k\). We know that \(\partial\eta_k/\partial\theta_1\equiv 1\) and \(\partial\eta_k/\partial\theta_2=k\). Then the derivatives with respect to \(\theta_1\) and \(\theta_2\) can be obtain with the chain rule: \[\begin{align*} \frac{df_k}{d\theta_i} &= \frac{\partial\eta_k}{\partial\theta_i} \frac{\partial f_k}{\partial\eta_k} \\ &= \frac{\partial \eta_k}{\partial \theta_i} \left[ y_k \frac{-e^{-\eta_k}}{1+e^{-\eta_k}} + (N-y_k) \frac{e^{\eta_k}}{1+e^{\eta_k}} \right] \\ &= \frac{\partial\eta_k}{\partial\theta_i} \left[ y_k \frac{-e^{-\eta_k/2}}{e^{\eta_k/2}+e^{-\eta_k/2}} + (N-y_k) \frac{e^{\eta_k/2}}{e^{-\eta_k/2}+e^{\eta_k/2}} \right] \\ &= \frac{\partial\eta_k}{\partial\theta_i} \left[ - y_k \frac{e^{-\eta_k/2}+e^{\eta_k/2}}{e^{\eta_k/2}+e^{-\eta_k/2}} + N \frac{1}{e^{-\eta_k}+1} \right] \\ &= \frac{\partial\eta_k}{\partial\theta_i} \left[ \frac{N}{e^{-\eta_k}+1} - y_k \right] \\ \end{align*}\] Since the derivatives of \(\eta_k\) with respect to \(\theta_1\) and \(\theta_2\) do not depend on the values of \(\theta_1\) and \(\theta_2\), the second order derivatives are \[\begin{align*} \frac{d^2f_k}{d\theta_i d\theta_j} &= \frac{\partial\eta_k}{\partial\theta_i} \frac{\partial\eta_k}{\partial\theta_j} \frac{\partial}{\partial\eta_k}\left[ \frac{N}{e^{-\eta_k}+1} - y_k \right] \\ &= \frac{\partial\eta_k}{\partial\theta_i} \frac{\partial\eta_k}{\partial\theta_j} \frac{N e^{-\eta_k}}{(e^{-\eta_k}+1)^2} \\ &= \frac{\partial\eta_k}{\partial\theta_i} \frac{\partial\eta_k}{\partial\theta_j} \frac{N}{(e^{-\eta_k/2}+e^{\eta_k/2})^2} \\ &= \frac{\partial\eta_k}{\partial\theta_i} \frac{\partial\eta_k}{\partial\theta_j} \frac{N}{4\cosh(\eta_k/2)^2} \\ \end{align*}\] Plugging in the \(\theta\)-derivatives gives the Hessain contribution for term \(f_k\) as \[ \frac{N}{4\cosh(\eta_k/2)^2} \begin{bmatrix}1 & k \\ k & k^2\end{bmatrix} . \]

Positive definite Hessian

To show that the total Hessian for \(f\) is positive definite for \(n \geq 2\), we define vectors \(\boldsymbol{u}_k=\begin{bmatrix}1 \\ k\end{bmatrix}\) and \(d_k=\frac{N}{4\cosh(\eta_k/2)^2}\). Define the 2-\(n\) matrix \(\boldsymbol{U}=\begin{bmatrix}\boldsymbol{u}_1 & \boldsymbol{u}_2 & \cdots & \boldsymbol{u}_n\end{bmatrix}\) and a diagonal matrix \(\boldsymbol{D}\) with \(D_{ii}=d_i\). The product \(\boldsymbol{U}\boldsymbol{D}\boldsymbol{U}^\top\) is then another way of writing the Hessian for \(f\). Since all the vectors \(\boldsymbol{u}_k\) are non-parallel, and the \(d_i\) are strictly positive for all combinations of \(\theta_1\) and \(\theta_2\), this matrix has full rank (rank 2) for \(n \geq 2\), and is positive definite.

Alternative reasoning

The Hessian for each \(f_k\) is positive semi-definite, since it can be written \(d_k\boldsymbol{u}_k\boldsymbol{u}_k^\top\) for some \(d_k > 0\) and vector \(\boldsymbol{u}_k\). The sum of a positive definite matrix and a positive semi-definite matrix is positive definite, so it’s sufficient to prove that the sum of the first two terms is positive definite. For any positive scaling constant \(w\), the determinant of \[ \begin{bmatrix}1 & 1\\1 & 1\end{bmatrix}+w\begin{bmatrix}1 & 2\\2 & 4\end{bmatrix} \] is \((1+w)(1+4w)-(1+2w)^2=1+5w+4w^2-1-4w-4w^2=w > 0\). this means that any (positively) weighted sum of those two matrices is positive definite (since each is positive semi-definite, and a positive determinant rules ot the sum being only positive semi-definite). This proves that the total Hessian is positive definite for \(n\geq 2\).

Remark

Note that in the first proof of positive definiteness, we didn’t actually need to know the specific values of the \(\boldsymbol{u}_k\) vectors, that are proportional to the gradients fo \(\eta_k\) with respect to \(\boldsymbol{\theta}=(\theta_1,\theta_2)\); it was sufficient that they were non-parallel, ensuring that \(\boldsymbol{U}\) had full rank. This means that any linear model for \(\eta_k\) in a set of parameters \(\boldsymbol{\theta}=(\theta_1,\dots,\theta_p)\) leads to a positive definite Hessian for this model, if the collection of gradient vectors of \((\eta_1,\dots,\eta_n)\) with respect to \(\boldsymbol{\theta}\) have collective rank at least \(p\).