Is All Whitening Core Safe On Colors

Permit your (centered) data be stored in a $n\times d$ matrix $\mathbf X$ with $d$ features (variables) in columns and $due north$ data points in rows. Let the covariance matrix $\mathbf C=\mathbf X^\top \mathbf X/n$ have eigenvectors in columns of $\mathbf E$ and eigenvalues on the diagonal of $\mathbf D$, then that $\mathbf C = \mathbf E \mathbf D \mathbf Due east^\top$.

And so what you call "normal" PCA whitening transformation is given by $\mathbf W_\mathrm{PCA} = \mathbf D^{-1/2} \mathbf Due east^\elevation$, see due east.grand. my respond in How to whiten the data using chief component analysis?

Still, this whitening transformation is not unique. Indeed, whitened data will stay whitened after any rotation, which ways that any $\mathbf W = \mathbf R \mathbf W_\mathrm{PCA}$ with orthogonal matrix $\mathbf R$ will besides be a whitening transformation. In what is chosen ZCA whitening, we have $\mathbf E$ (stacked together eigenvectors of the covariance matrix) equally this orthogonal matrix, i.e. $$\mathbf W_\mathrm{ZCA} = \mathbf E \mathbf D^{-1/2} \mathbf E^\top = \mathbf C^{-ane/2}.$$

I defining belongings of ZCA transformation (sometimes as well called "Mahalanobis transformation") is that information technology results in whitened data that is as close equally possible to the original data (in the least squares sense). In other words, if you lot want to minimize $\|\mathbf 10 - \mathbf Ten \mathbf A^\top\|^2$ subject to $ \mathbf 10 \mathbf A^\top$ beingness whitened, so y'all should take $\mathbf A = \mathbf W_\mathrm{ZCA}$. Here is a 2nd illustration:

Left subplot shows the data and its principal axes. Note the dark shading in the upper-correct corner of the distribution: it marks its orientation. Rows of $\mathbf W_\mathrm{PCA}$ are shown on the second subplot: these are the vectors the data is projected on. Subsequently whitening (below) the distribution looks circular, only notice that it also looks rotated --- nighttime corner is now on the East side, not on the North-E side. Rows of $\mathbf W_\mathrm{ZCA}$ are shown on the third subplot (note that they are non orthogonal!). Subsequently whitening (beneath) the distribution looks round and it's oriented in the same way equally originally. Of form, one tin can get from PCA whitened data to ZCA whitened data by rotating with $\mathbf E$.

The term "ZCA" seems to have been introduced in Bell and Sejnowski 1996 in the context of contained component analysis, and stands for "zero-stage component analysis". Run across there for more than details. Nigh probably, you came across this term in the context of image processing. It turns out, that when applied to a bunch of natural images (pixels every bit features, each epitome equally a data betoken), principal axes look similar Fourier components of increasing frequencies, see kickoff column of their Figure 1 beneath. So they are very "global". On the other hand, rows of ZCA transformation look very "local", see the second column. This is precisely because ZCA tries to transform the information as piddling every bit possible, and so each row should amend be shut to ane the original basis functions (which would be images with only one active pixel). And this is possible to achieve, because correlations in natural images are mostly very local (so de-correlation filters can besides exist local).

Update

More than examples of ZCA filters and of images transformed with ZCA are given in Krizhevsky, 2009, Learning Multiple Layers of Features from Tiny Images, see also examples in @bayerj's answer (+1).

I retrieve these examples give an idea as to when ZCA whitening might exist preferable to the PCA 1. Namely, ZCA-whitened images withal resemble normal images, whereas PCA-whitened ones expect nothing similar normal images. This is probably important for algorithms like convolutional neural networks (as east.g. used in Krizhevsky's paper), which treat neighbouring pixels together and so greatly rely on the local properties of natural images. For about other machine learning algorithms it should be admittedly irrelevant whether the information is whitened with PCA or ZCA.

Source: https://stats.stackexchange.com/questions/117427/what-is-the-difference-between-zca-whitening-and-pca-whitening

Posted by: kellermantinandeved.blogspot.com

Share this post