KalCal: a novel calibration framework for radio interferometry using the Kalman Filter and Smoother

Abstract

Calibration in radio interferometry is essential for correcting measurement errors. Traditional methods employ maximum likelihood techniques and non-linear least squares solvers but face challenges due to the data volumes and increased noise sensitivity of contemporary instruments such as MeerKAT. A common approach for mitigating these issues is using "solution intervals" , which helps manage the data volume and reduces overfitting. However, inappropriate interval sizes can degrade calibration quality, and determining optimal sizes is challenging, often relying on brute-force methods. This study introduces Kalman Filtering and Smoothing in Calibration (KalCal), a new framework for calibration that combines the Kalman Filter, Kalman Smoother, and the energy function: the negative logarithm of the Bayesian evidence. KalCal offers Bayesian-optimal solutions as probability densities and models calibration effects with lower computational requirements than iterative approaches. Unlike traditional methods, which require all the data for a particular solution to be in memory simultaneously, KalCal's recursive computations only need a single pass through the data with appropriate prior information. The energy function provides the means for KalCal to determine this prior information. Theoretical contributions include additions to complex optimisation literature and the "Kalman-Woodbury Identity" that reformulates the traditional Kalman Filter. A Python implementation of the KalCal framework was benchmarked against solution intervals as implemented in the QuartiCal package. Simulations show KalCal matching solution intervals in high Signal-to-Noise Ratio (SNR) scenarios and surpassing them in low SNR conditions. Moreover, the energy function produced minima that coincide with KalCal's Mean Square Error (MSE) on the true gain signal. This result is significant as the MSE is unavailable in real applications. Further research is needed to assess the computational feasibility and intricacies of KalCal.