Helge Rhodin

When people cross the street, they make a series of movements that are indicativeof their attention, intention, and comprehension of the roadside environment.These patterns in attention are linked to individual characteristics which are oftenneglected by autonomous vehicle prediction algorithms. We make two strides towardsmore personalized pedestrian modelling. First, we design an outdoor datastudy to collect behavioural signals such as pupil, gaze, head, and body orientationfrom an ego-centric human point of view. We gather this data over a rangeof diverse variables including age, gender, geographical context, crossing type,time of day, and the presence of a companion. In order for simulation engines tobe able to leverage such dense data, efficient 3D human and scene reconstructionalgorithms must be available. The increased resolution and model-free nature ofNeural Radiance Fields for large scene reconstruction and human motion synthesiscome at the cost of high training times and excessive memory requirements. Littlehas been done to reduce the resources required at training time in a manner thatsupports both dynamic and static tasks. Our second contribution takes the formof an efficient method which provides a reduction of the memory footprint, improvedaccuracy, and reduced amortized processing time both during training andinference. We demonstrate that the conscious separation of view-dependent appearanceand view-independent density estimation improves novel view synthesisof static scenes as well as dynamic human shape and motion. Further, we show thatour method, UNeRF, can be used to augment other state-of-the-art reconstructiontechniques to further accelerate and enhance the improvements which they present.

View record

Sensory substitution can help persons with perceptual deficits. In this work, we attempt to visualize audio with video. Our long-term goal is to create sound perception for hearing impaired people, for instance, to facilitate feedback for training deaf speech. Different from existing models that translate between speech and text or text and images, we target an immediate and low-level translation that applies to generic environment sounds and human speech without delay. No canonical mapping is known for this artificial translation task. Our design is to translate from audio to video by compressing both into a common latent space with a shared structure. Our core contribution is the development and evaluation of learned mappings that respect human perception limits and maximize user comfort by enforcing priors and combining strategies from unpaired image translation and disentanglement. We demonstrate qualitatively and quantitatively that our AudioViewer model maintains important audio features in the generated video and that generated videos of faces and numbers are well suited for visualizing high-dimensional audio features since they can easily be parsed by humans to match and distinguish between sounds, words, and speakers.

View record

In this thesis, we develop Computer Vision methods for Human body pose and stride length estimation. We first describe a framework for estimating the stride length of a walking subject from video using a multi-view camera setup. We specifically look into its utility in diagnosing Parkinson's disease. We do this using per frame 3D pose estimates and using an analysis of foot movement, we determine the length of the stride. Parkinson's diagnosis partly relies on stride length information; we claim that our method can be helpful in diagnosis. The current practice in the medical field is to estimate stride with complicated and fundamentally flawed sensors as they tend to affect the gait of the subjects using them. A benefit of our method is that cameras are relatively cheap, easily obtainable, and only need to be set up once. We also describe work done in improving the state of the art in human pose estimation. We first propose a pose refinement method that enhances state-of-the-art methods. Through analysis of our refiner, we show a flaw inherent in the human body model---the inaccuracy in the typical shape-to-pose regressor (joint regressor)---for a standard human pose dataset and show that the results on the top methods are actually being underreported. This flaw results in a situation where the ground truth joints are unsatisfiable with biologically plausible poses. We then address this flaw by modifying a part of the human body model. We reevaluate top state-of-the-art methods and show these models perform better with this modification without retraining.

View record

Navigating and sensing the world through echolocation in air is an innate ability in many animals for which analogous human technologies remain rudimentary. Many engineered approaches to acoustic reconstruction have been devised which typically require unwieldy equipment and a lengthy measurement process, and are largely not applicable in air or in everyday human environments. Recent learning-based approaches to single-emission in-air acoustic reconstruction use simplified hardware and an experimentally-acquired dataset of echoes and the geometry that produced them to train models to predict novel geometry from similar but previously-unheard echoes. However, these learned approaches use spatially-dense representations and attempt to predict an entire scene all at once. Doing so requires a tremendous abundance of training examples in order to learn a model that generalizes, which leaves these techniques vulnerable to over-fitting.We introduce an implicit representation for learned in-air acoustic reconstruction inspired by synthetic aperture focusing techniques. Our method trains a neural network to relate the coherency of multiple spatially-separated echo signals, after accounting for the expected time-of-flight along a straight-line path, to the presence or absence of an acoustically reflective object at any sampling location. Additionally, we use signed distance fields to represent geometric predictions which provide a better-behaved training signal and allow for efficient 3D rendering. Using acoustic wave simulation, we show that our method yields better generalization and behaves more intuitively than competing methods while requiring only a small fraction of the amount of training data.

View record