T-ESKF: Transformed Error-State Kalman Filter for Consistent Visual-Inertial Navigation

AI summary

Overview

Research area: Visual-inertial navigation systems (VINS), state estimation and simultaneous localization and mapping (SLAM), with a focus on estimator consistency in Kalman-filter-based fusion.

Technical level: Advanced. The paper relies on Lie group theory (SO(3), SE₂(3), SOT(3)), error-state Kalman filtering, observability analysis via the local observability matrix, and covariance propagation algebra.

Scope in one sentence: The paper introduces a linear time-varying transformation of the error state that makes the unobservable subspace of a visual-inertial estimator state-independent, yielding a consistent estimator called T-ESKF together with an efficient covariance propagation scheme.

What This Paper Is About

Visual-inertial navigation systems that fuse an IMU and a camera have four directions that cannot be observed without absolute information: global translation and rotation about the gravity direction. The standard Error-State Kalman Filter (ESKF) linearizes its transition and measurement Jacobians at different state estimates, which makes the rotation-about-gravity direction falsely appear observable, so the filter claims spurious information and becomes inconsistent. The authors' goal is to remove that mismatch by transforming the error-state system itself so that its unobservable subspace no longer depends on the state, and then to build a practical estimator (T-ESKF) around the transformed system without paying a heavy computational price.

Key Contributions

1. A transformation-based fix for VINS inconsistency. The authors design a linear time-varying transformation applied to the error state so that the unobservable subspace of the transformed system is independent of the state, and therefore unaffected by changes in the linearization point.

2. T-ESKF, a consistent VINS estimator. State estimation is performed using the transformed linearized error-state system, with covariance stored in the transformed space and mapped back to the original error-state space via an inverse transformation.

3. An efficient covariance propagation technique. Derived from the basis definition of the transformation, it expresses the transition and accumulated noise matrices of T-ESKF in terms of fixed-size 15×15 IMU matrices plus sparse transformation multiplications, avoiding repeated multiplications of size (15+3m)×(15+3m) where m is the number of landmarks. The authors state this technique is also applicable to RI-EKF.

4. An analytical proof of correct observability. The paper shows that because the discrete Jacobians are evaluated at the current best estimates, the optimality of the Jacobians is preserved, and the unobservable subspace of T-ESKF matches the correct four-dimensional one, so erroneous reductions in unobservable dimensions are prevented.

Main Findings

• Transformation removes state dependence: The chosen transformation uses T_p = [p̂]×, T_v = [v̂]×, T_l = [l̂]×, which turns the state-dependent blocks of the transition and essential measurement Jacobians into constant blocks (the transformed transition matrix contains [g]×, and the transformed essential measurement Jacobian becomes [0, −I₃, 0, I₃]).

• Correct observability is restored: The transformed system's unobservable subspace (Eq. 40) and the T-ESKF unobservable subspace (Eq. 53) are identical and state-independent, containing the global position directions and the gravity-direction rotation direction, unlike the ESKF subspace which drops the rotation direction and makes it falsely observable.

• Simulation accuracy (Table II, orientation in degrees / position in meters, average over 100 Monte-Carlo runs):

  • Udel-Gore: ESKF 1.13 / 0.26, FEJ-ESKF 0.67 / 0.21, RI-EKF 0.58 / 0.20, T-ESKF 0.58 / 0.20, Robo-Centric 0.59 / 0.21.
  • Udel-Neig.: ESKF 11.6 / 72.7, FEJ-ESKF 24.3 / 153, RI-EKF 6.92 / 42.2, T-ESKF 6.83 / 41.8, Robo-Centric 6.30 / 39.2.
  • TUM-Corr.: ESKF 0.68 / 0.28, FEJ-ESKF 0.36 / 0.16, RI-EKF 0.34 / 0.15, T-ESKF 0.34 / 0.15, Robo-Centric 0.36 / 0.15.
  • The paper notes Udel-Neig. is much longer than the other two trajectories and therefore shows larger RMSE values, and that FEJ-ESKF fails in most runs on Udel-Neig. primarily due to a larger first-order linearization error from poor feature initialization.

• Robustness to measurement noise: With visual noise levels of [0.4, 0.7, 1, 2, 3, 4, 5] pixels over 100 Monte-Carlo runs on Udel-Gore, T-ESKF shows better accuracy than FEJ-ESKF. Because FEJ-ESKF linearizes at initial estimated landmark positions, its accuracy degrades as noise grows, whereas T-ESKF uses the current best estimates and keeps Jacobian optimality.

• Consistency (NEES over 1000 Monte-Carlo runs on Udel-Gore): Except for ESKF, the average NEES of the compared estimators approaches 1; T-ESKF shows NEES values closer to 1 than FEJ-ESKF. The NEES histograms of T-ESKF and RI-EKF closely match the theoretical chi-square distribution. In ESKF the orientation error around the z-axis exceeds the ±3σ bounds, reflecting the falsely observable rotation.

• T-ESKF matches RI-EKF numerically: The paper states the two coincide in results (their plotted lines coincide), which the authors explain by the fact that T-ESKF and RI-EKF share an identical error-state system.

• Robo-Centric behavior: Robo-Centric shows slightly conservative orientation estimation, which the authors speculate may come from estimating gravitational direction in the local frame, coupling gravity and orientation; its orientation estimates around the x and y axes are more conservative than T-ESKF's.

• Computational efficiency: T-ESKF achieves computational efficiency comparable to ESKF while maintaining consistency (average time per frame on the Udel trajectory, on an R9 7950X @ 4.5GHz). For RI-EKF, Naive-RI (direct propagation) incurs substantial overhead in covariance propagation, while DES-RI and RI-EKF* (using the proposed technique) improve efficiency significantly.

• Real-world datasets: The authors report testing on EuRoC and TUM-VI with default OpenVINS parameters, with absolute trajectory error in Table III; the numerical values of that table are not present in the available text, so they are not reported here. Additional experiments on a customized sensor platform are stated to be in the supplemental material.

Methodology in Plain English

The estimator keeps an estimate of orientation, position, velocity, and landmark positions. Instead of estimating directly with the raw error state (the small difference between the true state and the estimate), the authors multiply that error state by a carefully chosen 12×12 transformation matrix that depends on the current estimate. The transformation is lower triangular with identity blocks on the diagonal, and its off-diagonal blocks are the skew-symmetric matrices of the estimated position, velocity, and landmark positions.

The purpose of the transformation is to cancel out the terms in the transition and measurement Jacobians that grow with the state. In the original ESKF, one block of the transition matrix contains the estimated rotation times the accelerometer measurement, and one block of the measurement Jacobian contains the estimated landmark minus estimated position. These are exactly the state-dependent pieces that make the unobservable direction move when the filter linearizes at different points. After transformation, those blocks become constants, and the observability matrix's null space no longer contains state estimates, so it survives changes in the linearization point.

To stay fast, the authors observe that their transformation relates the T-ESKF transition and accumulated noise matrices to the standard 15×15 IMU-only matrices of ESKF. This lets them compute the large landmark-coupled matrices by multiplying a fixed-size 15×15 block by sparse transformation matrices, rather than repeatedly multiplying matrices whose size grows with the number of landmarks. The estimator then propagates and updates covariance in the transformed space, computes the Kalman gain as in a classical EKF, transforms the state correction back to the original space, and corrects the state estimate.

Why This Matters

Impact on research. The work reframes the inconsistency problem in VINS as an error-state transformation problem rather than a state-redefinition problem. The authors explicitly contrast this with Lie group-based methods (RI-EKF, Equivariant Filter) and Robo-Centric formulations, which achieve consistency by redefining the state on manifolds. The paper argues that only the unobservable subspace of the transformed system needs to be state-independent, which opens a broader design space of possible transformations (Remark 1). It also connects its efficient propagation technique to RI-EKF by showing both share the same linearized error-state system, and shows that T-ESKF retains optimal Jacobians at the current best estimates, unlike FEJ, which fixes Jacobians at initial estimates.

Real-world applications:

• Robotics navigation, where compact and low-cost camera-IMU setups are used for pose estimation.
• Virtual reality and augmented reality headsets, which need reliable pose tracking from cameras and IMUs.
• Mobile and aerial platforms that must maintain trustworthy position and orientation estimates over long trajectories such as the Udel-Neig. sequence in the study.
• Visual-inertial SLAM pipelines that need statistically consistent uncertainty estimates for downstream sensor fusion.

Industry relevance. The estimator is implemented on OpenVINS, a widely used open-source filter-based VINS platform, and the code is released at github.com/HITCSC/T-ESKF. Competitive accuracy with ESKF-level runtime is directly relevant to embedded systems where optimization-based methods may be too costly, and consistency matters when the pose estimate feeds safety- or control-critical decisions.

Future Directions

• Exploring other transformation designs. The paper's Remark 1 states that making the Jacobians state-independent is not the only way to obtain a state-independent unobservable subspace, and that various solutions can be further explored.
• Extending the efficient propagation technique. The technique is stated to be applicable to RI-EKF, suggesting further application to other Lie group-based filters such as the Equivariant Filter, whose landmark handling differs.
• Full reporting of real-world results. The EuRoC and TUM-VI automated trajectory error table is referenced but not present in the available text, and experiments on the authors' customized sensor platform are deferred to supplemental material; these are the natural next validations to examine.
• Handling IMU bias explicitly. The model deliberately excludes IMU bias from the state to keep the formulation concise, with the bias-inclusive model placed in the supplemental material, so the full filter behavior with bias states remains an open evaluation area.

Target Audience

This paper is best suited to graduate students, postdoctoral researchers, and engineers who already work with Kalman filtering and Lie group state representations for visual-inertial odometry and SLAM. Readers implementing VINS on OpenVINS or similar platforms, and those interested in estimator consistency, observability analysis, and computational efficiency of filter-based navigation, will benefit most. A reader without a background in error-state Kalman filters and matrix observability analysis will need substantial additional reading to follow the derivations.

Authors’ abstract

This paper presents a novel approach to address the inconsistency problem caused by observability mismatch in visual-inertial navigation systems (VINS). The key idea involves applying a linear time-varying transformation to the error-state within the Error-State Kalman Filter (ESKF). This transformation ensures that \textrr{the unobservable subspace of the transformed error-state system} becomes independent of the state, thereby preserving the correct observability of the transformed system against variations in linearization points. We introduce the Transformed ESKF (T-ESKF), a consistent VINS estimator that performs state estimation using the transformed error-state system. Furthermore, we develop an efficient propagation technique to accelerate the covariance propagation based on the transformation relationship between the transition and accumulated matrices of T-ESKF and ESKF. We validate the proposed method through extensive simulations and experiments, demonstrating better (or competitive at least) performance compared to state-of-the-art methods. The code is available at github.com/HITCSC/T-ESKF.