EgoGVAE: Ego-body Mesh Reconstruction
via Guided Variational Autoencoder

We address the problem of recovering the full-body mesh from only the head pose. This task has become essential for various applications based on head-mounted devices or smart glasses. The challenge of this task lies in estimating the pose information of unobserved body parts based solely on a single joint (i.e., head) trajectory. Several studies have begun to adopt head-conditioned generative models; however, such previous methods are costly and time-consuming due to the diffusion-based iterative process.

As an alternative, we propose a simple yet novel method that leverages the latent space of the guidance network, which is designed as a variational autoencoder taking full-body poses as inputs. By enforcing latent distributions of this guidance network and our head-to-motion network to be similar, latent features sampled from the ‘guided’ distribution can be reliably decoded for natural representations of full-body poses even only with the head pose. One important advantage of the proposed method is that the one-step sampling scheme achieves remarkably fast inference (more than 50 times faster) compared to diffusion-based approaches. Experimental results on benchmark datasets show that the proposed method efficiently improves the performance of ego-body mesh reconstruction.

We propose a simple yet powerful method, EgoGVAE, for full-body mesh reconstruction from only the head pose of the wearer. EgoGVAE leverages the latent space of the motion-to-motion network, which is a variational autoencoder that takes full-body poses as inputs, to guide the head-to-motion network. This design scheme, which operates with one-step sampling in inference, makes EgoGVAE perform very fast compared to diffusion-based approaches.