Kwang Moo Yi

Neural radiance fields (NeRF) have brought progress in rendering photorealistic 3D reconstruction. However, it requires clear images with correct camera poses. To address this problem, we propose to model camera imperfections that arise from the simple pinhole camera model and combine RGB images with event camera data in a stereo setup. Specifically, compared to conventional approaches that enforce physical priors on a camera model, we model measurement variation across the exposure time using embeddings using a data-driven approach. To incorporate event data into the NeRF pipeline, we propose a learnable mapper that bridges the event camera measurement space with that of the RGB camera. To validate our method, we collected our own high-resolution RGB and event stereo dataset. For further validation, we utilize the EVIMOv2 dataset consisting of limited indoor scenes.

View record

Knowledge of galaxy distance is important for cosmological studies. Recent deep learning-based approaches may not leverage the full potential of the neural network. We propose a generative model to reconstruct 1D electromagnetic spectra with application to estimate astronomical redshift. The generative model is an auto-decoding neural field network. We represent each spectrum as a high-dimensional embedding which is converted to spectra reconstruction by the following decoder. We optimize the decoder in restframe simultaneously with the embedding by maximizing the structural similarity between the reconstructed and the observed spectra. We then train a classifier based on the reconstructed spectra for redshift classification. During inference, we fit the auto-decoder to the test spectra and then use the classifier to estimate redshift. Compared with a regressor, our classification model features a simplified optimization surface. We combine spectroscopic data from the zCOSMOS , the DEIMOS , and the VIPERS surveys as our dataset. We split the data into training and testing data and outperform the baseline by 0.6% on test redshift accuracy.

View record

Enabling computers to understand and interpret visual information is crucial for the development of more sophisticated and interactive technologies. Learning structure, such as 3D shape, can help computers understand visual information more effectively. Our model disentangles object structure and appearance in a weakly supervised manner from multiview images within a single category. We extract a 3D pointcloud from images and reconstruct consistent novel views by rendering the pointclouds from different perspectives. Using a much simpler model and far fewer training examples than costly state-of-the-art diffusion models, we can recover 3D structure from single images of objects and quickly reconstruct them from unseen viewpoints. Our findings suggest that understanding 3D constraints of the real world can enhance performance on visual tasks and make models more robust and generalizable to a wider variety of inputs. This approach enables downstream tasks such as pose transfer, spatially-guided conditional image generation, and paves the way for commonsense reasoning. Our work has potential applications in augmented reality and visual effects, with further exploration of the model's capabilities and integration into broader systems for enhanced visual understanding.

View record

We present a novel method to provide efficient and highly detailed reconstructions. Inspired by wavelets, we learn a neural field that decompose the signal both spatially and frequency-wise. We follow the recent grid-based paradigm for spatial decomposition, but unlike existing work, encourage specific frequencies to be stored in each grid via Fourier features encodings. We then apply a multi-layer perceptron with sine activations, taking these Fourier encoded features in at appropriate layers so that higher-frequency components are accumulated on top of lower-frequency components sequentially, which we sum up to form the final output. We demonstrate that our method outperforms the state of the art regarding model compactness and convergence speed on multiple tasks: 2D image fitting, 3D shape reconstruction, and neural radiance fields. Our code is available at https://github.com/ubc-vision/NFFB.

View record

In this work, we propose a bootstrapping framework to enhance human optical flow and 3D human pose. We show that, for videos involving humans in scenes, we can improve both the optical flow and the pose estimation quality of humans by considering the two tasks at the same time. Generic optical flow methods perform better on humans when fine-tuned on human-centric scenes showing that the focus should be on humans when the task is human oriented. On the other hand, an overlooked assumption in recent 3D human pose estimation methods is temporal consistency. As such, we make use of existing human pose estimators and optical flow networks and improve their performance by benefitting from each other. In more detail, we optimize the pose and optical flow networks to, at inference time, agree with each other. We show that this results in state-of-the-art performance on the Human 3.6M and 3D Poses in the Wild datasets, as well as a human-related subset of the Sintel dataset, both in terms of pose estimation accuracy and the optical flow accuracy at human joint locations.

View record

In this thesis, we develop Computer Vision methods for Human body pose and stride length estimation. We first describe a framework for estimating the stride length of a walking subject from video using a multi-view camera setup. We specifically look into its utility in diagnosing Parkinson's disease. We do this using per frame 3D pose estimates and using an analysis of foot movement, we determine the length of the stride. Parkinson's diagnosis partly relies on stride length information; we claim that our method can be helpful in diagnosis. The current practice in the medical field is to estimate stride with complicated and fundamentally flawed sensors as they tend to affect the gait of the subjects using them. A benefit of our method is that cameras are relatively cheap, easily obtainable, and only need to be set up once. We also describe work done in improving the state of the art in human pose estimation. We first propose a pose refinement method that enhances state-of-the-art methods. Through analysis of our refiner, we show a flaw inherent in the human body model---the inaccuracy in the typical shape-to-pose regressor (joint regressor)---for a standard human pose dataset and show that the results on the top methods are actually being underreported. This flaw results in a situation where the ground truth joints are unsatisfiable with biologically plausible poses. We then address this flaw by modifying a part of the human body model. We reevaluate top state-of-the-art methods and show these models perform better with this modification without retraining.

View record