Implementing Augmented Reality: Mapping GPS Coordinates to Screen Pixels

From GPS Coordinates to Screen Pixels: The Full AR Projection Math

Developer Dhruv Anand detailed a four-stage coordinate transformation pipeline for overlaying real-time ADS-B aircraft data on Android camera feeds. The system accounts for Earth’s curvature and sensor fusion to map geodetic coordinates to specific screen pixels.

Why This Matters

In AR development, ‘plausible’ math often fails because developers neglect the divergence of meridians or sign-flip conventions between device and camera frames. At 24.86 degrees N latitude, omitting the cosine correction in longitude results in a 9.3 percent positional error, leading to misaligned overlays that degrade user experience despite compiling correctly. Practical constraints like sensor latency and lens distortion add further complexity, where a fast 60 degrees per second pan can introduce a 16-pixel error, necessitating low-pass filtering for stability.

Key Insights

• Stage 1 Geodetic-to-ENU: Uses a local tangent plane approximation valid for distances under 100 km, where relative error is restricted to ~0.11% (approx. 100m).
• Cosine Correction: Longitude displacement must be scaled by cos(latitude) because meridians converge at the poles; failure to do so creates increasing East-West errors at higher latitudes.
• Matrix Orthogonality: Android’s TYPE_ROTATION_VECTOR matrix is orthonormal, allowing the inverse transform to be calculated via a simple transpose (R transpose).
• Axis Convention Shift: Transitioning from device frame to camera frame in portrait mode requires a Z-axis sign flip (Cz = -dZ) to account for the lens pointing away from the user.
• Perspective Divide: Normalizing coordinates by the tangent of the half-FOV maps the visible 3D frustum to Normalized Device Coordinates (NDC) between -1 and 1.
• Magnitude Preservation: Because rotation matrices are orthonormal, the magnitude of the device frame vector must match the original ENU vector magnitude as a sanity check.

Working Examples

Logic for classifying off-screen aircraft positions and placing edge indicators.

if (Cz <= 0) {
    // Aircraft is behind the camera
    showIndicator(BEHIND);
} else if (abs(NDCx) > 1 || abs(NDCy) > 1) {
    // Off-screen classification
    if (NDCx < -1) x = p, y = clamp(ypx, p, H-p); // LEFT
    if (NDCx > 1) x = W-p, y = clamp(ypx, p, H-p); // RIGHT
    if (NDCy > 1) x = clamp(xpx, p, W-p), y = p; // UP
    if (NDCy < -1) x = clamp(xpx, p, W-p), y = H-p; // DOWN
}

Practical Applications

• ADS-B tracking apps use the Haversine formula for exact distances beyond 100 km to maintain label accuracy. Pitfall: Using flat-earth math for long-range targets results in significant pixel drift.
• AR navigation systems implement the R-transpose matrix multiplication for magnitude preservation checks. Pitfall: Incorrect row-major indexing in the FloatArray(9) leads to non-orthogonal transformations and distorted projections.
• Camera overlay systems apply the (1 - NDCy) / 2 transformation to align camera space with screen pixel space. Pitfall: Neglecting the Y-axis flip causes objects above the horizon to appear below the screen center.

References:

• https://dev.to/ananddhruv29/from-gps-coordinates-to-screen-pixels-the-full-ar-projection-math-4k7m