Probabilistic foundations of Kernel Density & Bandwidth.
Kernel Density Estimation (KDE) is a cornerstone of non-parametric statistics, allowing analysts to infer the underlying probability density function of a continuous random variable without imposing rigid parametric family assumptions. The accuracy of kernel density estimation hinges upon the selection of the bandwidth smoothing parameter h, which arbitrates a classical trade-off between variance and bias. In this laboratory, we simulate empirical sample datasets, evaluate Gaussian kernel approximations, explore automated bandwidth estimators, and experimentally verify the fundamental N^(-1/5) asymptotic power law governing optimal bandwidth convergence. In mathematical statistics, empirical data science, and modern computational engineering, understanding the asymptotic sampling properties and theoretical behavior of Kernel Density Estimation and Bandwidth Optimization via Silverman's Rule provides crucial guarantees. Rigorous experimental validation connects foundational probability laws with observable empirical phenomena, ensuring that statistical inference remains valid under finite sample sizes and real-world data collection constraints.
Viva voce questions & analytical theory
What is the mathematical formulation of the Kernel Density Estimator (KDE)?
Let X_1, ..., X_N be an i.i.d. sample from density f. The kernel density estimator is f̂_h(x) = (1 / (N h)) ∑_{i=1}^N K((x - X_i) / h), where K(u) is a non-negative kernel function integrating to 1 with zero mean and finite variance, and h > 0 is the smoothing bandwidth. It places a kernel bump over each observation to produce a smooth, continuous density reconstruction.
How is the Mean Integrated Squared Error (MISE) decomposed into bias and variance?
MISE(h) = E[∫ (f̂_h(x) - f(x))^2 dx] = ∫ Bias^2(f̂_h(x)) dx + ∫ Var(f̂_h(x)) dx. Using Taylor expansions for small h, asymptotic squared bias is (1/4) h^4 μ_2(K)^2 ∫ (f''(x))^2 dx, while asymptotic variance is (1 / (N h)) ∫ K(u)^2 du. Thus MISE(h) ≈ (1/4) h^4 μ_2(K)^2 R(f'') + (1 / (N h)) R(K), demonstrating the fundamental bias-variance trade-off.
How is Silverman's rule of thumb derived mathematically from MISE minimization?
Differentiating AMISE(h) with respect to h and setting to zero: d(AMISE)/dh = h^3 μ_2(K)^2 R(f'') - (1 / (N h^2)) R(K) = 0 yields h_AMISE = [R(K) / (μ_2(K)^2 R(f'') N)]^(1/5) ∝ N^(-1/5). For a Gaussian kernel and Gaussian target N(0, σ^2), evaluating roughness integrals yields Silverman's formula h_opt = (4σ^5 / 3N)^(1/5) ≈ 1.059 σ N^(-1/5), or with robust scale A = min(s, IQR/1.349), h_opt = 0.9 A N^(-1/5).
What happens when bandwidth is undersmoothed versus oversmoothed?
Undersmoothing (h ≪ h_opt) makes variance dominate: the density exhibits spurious noise spikes and artificial modes centered on individual data points. Oversmoothing (h ≫ h_opt) makes bias dominate: genuine multimodal peaks are flattened, valleys are bridged, and mass is artificially smeared into the tails.
How much does the choice of kernel function influence estimation efficiency?
The Epanechnikov kernel minimizes AMISE, but Epanechnikov showed that standard alternatives like Gaussian, biweight, and triangular kernels achieve over 95% relative efficiency. Kernel choice has minor practical impact, whereas bandwidth selection is critical.
How does Least Squares Cross-Validation (LSCV) select bandwidth in a data-driven manner?
LSCV minimizes Integrated Squared Error without knowing true f by minimizing LSCV(h) = ∫ f̂_h(x)^2 dx - (2/N) ∑_{i=1}^N f̂_{h,-i}(X_i), using leave-one-out kernel estimates. Minimizing LSCV(h) provides an objective data-driven bandwidth.
Why does KDE suffer from the curse of dimensionality in multivariate settings?
In d dimensions, optimal bandwidth scales as h_opt ∝ N^(-1/(d+4)), and optimal MISE scales as O(N^(-4/(d+4))). For d = 1, error decays as N^(-0.8); for d = 10, error decays as N^(-0.286), requiring exponentially larger samples to maintain accuracy.
Experimental precautions and statistical checks
- Ensure the evaluation grid has at least 200 points to prevent visual stepping artifacts.
- Apply boundary reflection when estimating densities on bounded supports (e.g. x ≥ 0) to eliminate boundary bias.
- Verify kernel functions integrate numerically to 1.0 to preserve total probability.
- Perform replication across multiple independent pseudorandom seeds to verify that estimator variance and bias bounds are invariant to initial generator states.
Theoretical assumptions & limitations
- Fixed-bandwidth KDE applies uniform smoothing everywhere, causing local oversmoothing or undersmoothing.
- Evaluation complexity is O(N · M) where M is grid size, requiring FFT methods for big data.