SRPose: Two-view Relative Pose Estimation With Sparse Keypoints

ECCV 2024

Rui Yin1, Yulun Zhang2, Zherong Pan3, Jianjun Zhu1, Cheng Wang1, Biao Jia1
1Hanglok-Tech, 2Shanghai Jiao Tong University 3Tencent America
arXiv Code
SRPose Poster SRPose Teaser

Fig. 1: Relative pose estimation by SRPose. Dots drawn in the figures visualize the cross-attention scores of sparse keypoints across the two images, with brighter dots representing higher attention. Camera-to-world: (a), (b), (c) visualize the epipolar lines, representing the connections between the nine corresponding points across two views. Higher attention is shown to the overlap of the scenes. Object-to-Camera: (d), (e), (f) show the relative 6D pose estimation in the query image with only one accessible object prompt in the reference image. Higher attention is shown to the target object. SRPose establishes implicit correspondences.

Abstract

Two-view pose estimation is essential for map-free visual relocalization and object pose tracking tasks. However, traditional matching methods suffer from time-consuming robust estimators, while deep learning-based pose regressors only cater to camera-to-world pose estimation, lacking generalizability to different image sizes and camera intrinsics. In this paper, we propose SRPose, a sparse keypoint-based framework for two-view relative pose estimation in camera-to-world and object-to-camera scenarios. SRPose consists of a sparse keypoint detector, an intrinsic-calibration position encoder, and promptable prior knowledge-guided attention layers. Given two RGB images of a fixed scene or a moving object, SRPose estimates the relative camera or 6D object pose transformation. Extensive experiments demonstrate that SRPose achieves competitive or superior performance compared to state-of-the-art methods in terms of accuracy and speed, showing generalizability to both scenarios. It is robust to different image sizes and camera intrinsics, and can be deployed with low computing resources.

Approach

SRPose Overview

Fig. 3: Overview. SRPose comprises four main components: 1) The sparse keypoint detector detects keypoints associated with descriptors separately sfrom the two images; 2) The intrinsic-calibration (IC) position encoder modulates the keypoints' coordinates with camera intrinsics, and encodes their position information; 3) Guided by the prior knowledge of keypoint similarities, along with the object prompt, the attention layers establish implicit cross-view correspondences; 4) The regressor estimates relative pose R, t under the constraints of implicit correspondences.

BibTeX

@inproceedings{yin2024srpose,
    title={SRPose: Two-view Relative Pose Estimation with Sparse Keypoints},
    author={Yin, Rui and Zhang, Yulun and Pan, Zherong and Zhu, Jianjun and Wang, Cheng and Jia, Biao},
    booktitle={ECCV},
    year={2024}
}