Dinesh Pai

Deep generative models such as Variational Autoencoders (VAEs), Generative Ad- versarial Networks (GANs), and diffusion models have demonstrated their efficacy in generating 2D images and 3D meshes. However, interpreting the learned latent space poses a significant challenge. The current literature mainly focuses on un- supervised methods, which exhibit two primary limitations: firstly, the inability to control the meaning of each latent variable, and secondly, the occurrence of multiple latent variables possessing overlapping meanings. Moreover, it has been shown that fully disentangling the latent space using only unsupervised methods is theoretically infeasible. In this work, we introduce a method for latent space disen- tanglement on 3D meshes. Our method comprises two components: a feature func- tion for predicting 3D mesh features, and a regular generative model. We employ the derivative of the feature function as part of the loss function to promote dis- entanglement. Experimental results demonstrate that our disentanglement method effectively addresses the limitations mentioned above without compromising the accuracy of the reconstruction. Additionally, given its model-agnostic nature, our method exhibits generality across different generative models and tasks.

View record

Real-time animations are limited by a computational budget and often trade realism for performance. Simulating the shape of a realistic human body requires a tremendous amount of computational resources and may be infeasible in real-time applications. In recent years, deep learning approaches have proven their effectiveness in fields such as computer vision. We present a new method that combines ideas from deep learning and example-based skinning. The method approximates corrections to skin deformation from a skeleton-based animation baseline. A key aspect of the approach is to factor the network into two parts, with part of the network evaluated using shaders in the standard real-time graphics rendering pipeline. Our method adds a minimum overhead to a skeleton-based animation while improving its visual results.

View record

Fit and sizing of clothing are fundamental problems in the field of garment design, manufacture and retail. Here we propose new computational methods for adjusting the fit of clothing on realistic models of the human body by interactively modifying desired fit attributes. Clothing fit represents the relationship between the body and the garment, and is quantified using fit attributes such as ease and pressure on the body. Such attributes are computed by physically based simulations. We propose a method to learn the relationship between the fit attributes and the space of pattern edits. In contrast to the earlier approaches that use in-the-loop physics simulation, we begin by creating a custom data set capturing all possible edits to the 2D pattern and the corresponding per-vertex fit attributes generated from their 3D drape simulations. With this data we train a model to isolate and predict changes to the 2D pattern caused by edits to the fit attributes. We provide interactive tools to directly edit the fit attributes in 3D and instantaneously predict the corresponding pattern adjustments. Our method introduces a different way to express fit adjustment that it is more intuitive and is capable of cutting short the trial-and-error period.

View record

In this thesis, we address the challenge of acquiring high-quality measurement data from real-world experiments. Experiments with human participants can be expensive, both in terms of scheduling participants as well as equipment requirements. Because of these constraints limiting data collection, we desire software tools for data quality assurances that are active during each measurement session. We propose a modular approach to conducting experiments based on the inputs, outputs, and dependencies between individual data-generating operations that we call measurement services. Formally defining the output of each operation provides clear quality assurance targets to aim for during the experiment session. Our framework of modular components also emphasizes extensibility and reusability in the development of new experiments. We implemented our approach by developing BodyData, a web application-centered system designed to measure, store, and securely access data from experiments with human participants. BodyData was tested in our lab; two case studies are presented to illustrate the utility of the system in practice. We discuss how we provide improved quality assurance through the use of configurable data entry constraints as well as visual feedback during the measurement session. We also discuss how we support queries from authorized clients for use in analysis and visualization of stored data.

View record

In cloth simulation, the choice of material parameters drive the motion of cloth. A good cloth simulation resembles the real world appearance as best as possible. Functional garments as a whole are inhomogeneous, though every distinct part is homogeneous at a small scale. Here, we measure all individual parts of a sports bra and characterize their material parameters.I build a custom designed cloth tester that is capable of measuring a variety of different cloth samples. In particular, common swathes of cloth, but also thicker and stiffer seams can be assessed. Force-displacement curves for both shear and stretch experiments are estimated. At the same time, visual deformation is tracked with a camera. I then simulate the same piece of cloth and minimize the difference between the simulated and the experimentally observed cloth sample to tune our material parameters. At the heart of our cloth simulation lies a non-linear and anisotropic material model.Results show that our device can handle all respective garment parts of a modern sports bra. From tests with synthetic data we learn that our optimization converges to all ground truth material parameters but the bending stiffness. For the measured sports bra, the estimated material parameters fall within the range of values of comparable materials. Again, the bending stiffness has minimal influence on the objective function and we can not resolve the true bending stiffness.In combination, the new measurement device and a cantilever test can estimate the material parameters shear, bulk and bending stiffness and the non-linear stress-strain curves. Thus, the individual garment pieces can be combined to simulate the whole garment.

View record

Personalized simulation of human bodies is a long standing goal in many applications, ranging from animation to apparel. Even though personalized geometric models can now be easily acquired, physics-based simulation requires soft tissue properties and their distribution over the person’s body. Here we show that mechanical properties of the human body can be directly measured using a novel hand-held device. We describe a complete pipeline for measurement, modeling, parameter estimation, and simulation. The methods described here can be used to create personalized models of an individual human or of a population. Furthermore, we show how to predict soft tissue properties from widely available 3D geometric models of the human body. To train such a prediction model, we utilize a unique database of registered measurements of body shape and soft tissue properties, acquired from over 70 participants. We use a recently introduced convolutional neural network architecture adapted for 3D surfaces, and train the network to predict the distribution of tissue properties over the 3D human body surface. Once the network is trained, no specialized equipment is required, and soft tissue properties are predicted in minutes. The method can be used with commodity 3D scanners, and even with geometric models downloaded from Internet or created by artists. Our methods make realistic human body simulations available to a wide range of users and applications.

View record

Simulation of human soft tissues in contact with their environment is essential in many fields, including visual effects and apparel design. Biological tissues are nearly incompressible. However, standard methods employ compressible elasticity models and achieve incompressibility indirectly by setting Poisson’s ratio to be close to 0.5. This approach can produce results that are plausible qualitatively but inaccurate quantatively. This approach also causes numerical instabilities and locking in coarse discretizations or otherwise pose a prohibitive restriction on the size of the time step. We propose a novel approach to alleviate these issues by replacing indirect volume preservation using Poisson’s ratios with direct enforcement of zonal volume constraints, while controlling fine-scale volumetric deformation through a cell-wise penalty. To increase realism, we propose an epidermis model to mimic the dramatically higher surface stiffness on real skinned bodies. We demonstrate that our method produces stable realistic deformations with precise volume preservation but without locking artifacts. Due to the volume preservation not being tiedto mesh discretization, our method also allows a resolution consistent simulation of incompressible materials.

View record

We present "SkinProbe 2.0," a prototype system for low cost, high volume measurement of the physical properties of human soft tissues through direct contact and perturbation of the skin. Our solution encompasses a handheld device and associated cloud-based AI processing pipeline, and derives physically-representative values of stiffness and thickness directly from video. These input videos include images of the surface under contact, and of a "flexure," our novel apparatus for optical force measurement. Videos are captured using a smartphone embedded in the device.Our system processes these videos, generating dense optical flow fields for selected frames, and passing these frames and flow fields through two bespoke Neural Networks: one providing estimated force readings, and one providing estimates of soft-body material properties in the contact vicinity.We automate the collection of training data for our networks with robotics and a 3D-printed apparatus, along with custom-made silicone tissue phantoms, and a cloud pipeline for data collection, storage, and retrieval. This allows us to scale to thousands of samples in each training dataset, with minimal human involvement in collection, and a highly repeatable collection process.We demonstrate the functionality of our measurement device, cloud pipeline, and force estimation system, and show promising material estimation results on our tissue phantoms. We further consider directions for future research in improving our system, both for handheld data collection, and for eventual usage on human subjects.

View record

Virtual Reality (VR) and related 3D display technologies have recently experienced tremendous growth in promise and popularity, but have significant limitations. Human vision, carefully tuned to integrating multiple cues from the real words can incorrectly perceive the virtual world in these displays. In this thesis, we conduct a series of psychophysics experiments evaluating motion perception in VR, culminating in a user-adapted method to increase interception accuracy of virtual objects by modifying motion-in-depth cues. Using a baseball hitting simulation in VR, we show that our modified motion-in-depth cues result in greater accuracy. Finally, we present implementations of 3D gaze analysis algorithms.

View record

Humans are extremely sensitive to facial realism and spend a surprisingly amount of time focusing their attention on other people's faces. Thus, believable human character animation requires realistic facial performance. Various techniques have been developed to capture highly detailed actor performance or to help drive facial animation. However, the eye region remains a largely unexplored field and automatic animation of this region is still an open problem. We tackle two different aspects of automatically generating facial features, aiming to recreate the small intricacies of the eye region in real-time. First, we present a system for real-time animation of eyes that can be interactively controlled using a small number of animation parameters, including gaze. These parameters can be obtained using traditional animation curves, measured from an actor’s performance using off-the-shelf eye tracking methods, or estimated from the scene observed by the character using behavioral models of human vision. We present a model of eye movement, that includes not only movement of the globes, but also of the eyelids and other soft tissues in the eye region. To our knowledge this is the first system for real-time animation of soft tissue movement around the eyes based on gaze input. Second, we present a method for real-time generation of distance fields for any mesh in screen space. This method does not depend on object complexity or shape, being only contained by the intended field resolution. We procedurally generate lacrimal lakes on a human character using the generated distance field as input. We present different sampling algorithms for surface exploration and distance estimation, and compare their performance. To our knowledge this is the first method for real-time or screen space generation of distance fields.

View record

Wrinkling of human skin significantly affects the realism of computer generated characters. Wrinkles convey emotion and expression, provide clues of age and health, and indicate interaction between the skin and external objects. Wrinkling is caused by compression: an elastic material buckles out-of-plane in order to preserve length and volume. Human skin buckles in a distinctive pattern, characterized by sharp valleys with rounded peaks. Many techniques used in visual effects require artists to directly produce wrinkles through sculpting or painted displacement maps, while automated techniques are generally designed for adding detail to coarse, cloth-like simulations which are usually not consistent with human skin. The layered structure of skin, and the properties of each layer are critical to producing the buckling patterns observed in real life.In this work a simulation of wrinkling skin is developed that is physically based, while also simple enough for use in computer graphics. A novel constitutive model suitable for large compressive strain is derived and applied to a three-layered model of skin, with a thin shell outermost layer (stratum corneum), and volumetric dermis and hypodermis layers. Finally, we present a modified Newton scheme and linear finite elements for simulating equilibrium configurations of skin under compressive strain.

View record

In ball sports, athletes are taught to keep their eyes on the ball to catch or hit it successfully. This intuitive field experience has already been studied in the laboratory, indicating that tracking a moving object with smooth pursuit eye movementsenhances our ability to predict the object’s trajectory in time and space. Similarly, intercepting a moving object critically relies on motion prediction. Here we assessed the functional significance of eye movements for manual interceptions.In a novel paradigm, we asked observers (n=32) to track a small moving dot, backprojected onto a translucent screen, and to intercept it with their index finger in a designated ‘hit zone’. Hereby, only the first part (100-300 ms) of the trajectory was shown. Thus, observers had to extrapolate the trajectory and intercept its assumed position anywhere within the hit zone.Results show that better pursuit (low eye position and velocity error, high velocity gain, few catch-up saccades of small amplitude) lead to more accurate interceptions. A Hazard analysis yielded two interception strategies: Early interceptors reliedon tracking quality and memory feedback given at the end of each trial, while late interceptors depended more on tracking smoothness, small initial saccades, and accurate eye latencies. Early interceptions (less time of invisibility) yieldedsmaller 2D interception error, while the interception timing was better for longerperiods of smooth tracking (later interceptions).A regression model tree identified low tracking error and small saccadic eye movements as those eye parameters predicting accurate interceptions best. Not only do observers benefit from smooth pursuit eye movements during manual interception, but the interception accuracy also scales with the quality of the eye movements.

View record

There have been many recent advances in the simulation of biologically realistic systems, but controlling these systems remains a challenge. In this thesis, we focus on methods for learning to control these systems without prior knowledge of the dynamics of the system or its environment.We present two algorithms. The first, designed for quasistatic systems, combines Gaussian process regression and stochastic gradient descent. By testing on a model of the human mid-face, we show that this combined method gives better control accuracy than either regression or gradient descent alone, and improves the efficiency of the optimization routine. The second addresses the trajectory-tracking problem for dynamical systems. Our method automatically learns the relationship between muscle activations and resulting movements. We also incorporate passive dynamics compensation and propose a novel gain-scheduling algorithm. Experiments performed on a model of the human index finger demonstrate that each component we add to the control formulation improves performance of fingertip precision tasks.

View record

Human observers tend to pay a lot of attention to the eyes and the surrounding soft tissues. These periorbital soft tissues are associated with subtle and fast motions that convey emotions during facial expressions. Modeling the complex movements of these soft tissues is essential for capturing and reproducing realism in facial animations.In this work, we present a data driven model that can efficiently learn and reproduce the complex motion of the periorbital soft tissues. We develop a system to capture the motion of the eye region using a high frame rate monocular camera. We estimate the high resolution texture of the surrounding eye regions using a Bayesian framework. Our learned model performs well in reproducing various animations of the eyes. We further improve realism by introducing methods to model facial wrinkles.

View record

Humans show stunning performance on a variety of manipulation tasks. However, little is known about the computation that the human brain performs to accomplish these tasks. Recently, anatomically correct tendon-driven models of the human hand have been developed, but controlling them remains an issue. In this thesis, we present a computationally efficient feedback controller, capable of dealing with the complexity of these models. We demonstrate its abilities by successfully performing tracking and reaching tasks for an elaborated model of the human index finger.The controller, called One-Step-Ahead controller, is designed in a hierarchical fashion, with the high-level controller determining the desired trajectory and the low-level controller transforming it into muscle activations by solving a constrained linear least squares problem. It was proposed to use equilibrium controls as a feedforward command, and learn the controller's parameters online by stabilizing the plant at various configurations. The conducted experiments suggest the feasibility of the proposed learning approach for the index finger model.

View record

When interacting with physical objects using their own hands, humans display effortless dexterity. It remains a non-intuitive task, however, to specify the motion of a virtual character’s hand or of a robotic manipulator. Creating these motions generally requires animation expertise or extensive periods of offline motion capture. This thesis presents a real-time, adaptive animation interface, specifically designed around haptic (i.e., touch) feedback, for creating precision manipulations of virtual objects. Using this interface, an animator controls an abstract grasper trajectory while the full hand pose is automatically shaped by compliant scene interactions and proactive adaptation. Haptic feedback enables intuitive control by mapping interaction forces from the full animated hand back to the reduced animator feedback space, invoking the same sensorimotor control systems utilized in natural precision manipulations. We provide an approach for online, adaptive shaping of the animated manipulator using our interface based on prior interactions, resulting in more functional and appealing motions.In a user study with nonexpert participants, we tested the effectiveness of haptic feedback and proactive adaptation of grasp shaping. Comparing the quality of motions produced with and without force rendering, haptic feedback was shown to be critical for efficiently communicating contact forces and dynamic events to the user. The effects of proactive shaping, though inarguably beneficial to visual quality, resulted in mixed behavior for our grasp quality metrics.

View record

This thesis presents a biomechanically based hand simulator. We makecontributions at two di erent levels: hand motion and hand appearance.We rst develop a musculotendon simulator, and apply this simulator toan anatomically based hand model. Anatomically based hand simulation ischallenging because the tendon network of the hand is complicated and itis highly constrained by the skeleton of the hand. Our simulator employsthe elegance of the Eulerian-Lagrangian strand algorithm, and introducesa 2D planar collision approach to e ciently eliminate unnecessary degreesof freedom and constraints. We show that with our method, we obtain thecoupling between joints automatically, and achieve the storage of energy intendons for fast movements. Also, by injuring a tendon, we are able toobtain simulations of common nger deformities.Although the musculotendon based hand simulation produces naturalhand motion, hand animation is usually observed at the skin level. Wepresent a novel approach to simulate thin hyperelastic skin. Real humanskin is a thin tissue which can stretch and slide over underlying body structuressuch as muscles, bones, and tendons, revealing rich details of a movingcharacter. Simulating such skin is challenging because it is in close contactwith the body and shares its geometry. We propose a novel Eulerian representationof skin that avoids all the di culties of constraining the skin to lieon the body surface by working directly on the surface itself. Skin is modeledas a 2D hyperelastic membrane with arbitrary topology, which makes it easyto cover an entire character or object. We use triangular meshes to modelbody and skin geometry. The method is easy to implement, and can use lowresolution meshes to animate high resolution details stored in texture-likemaps. Skin movement is driven by the animation of body shape prescribedby an artist or by another simulation, and so it can be easily added as apost-processing stage to an existing animation pipeline. We demonstraterealistic animations of the skin on the hand using this approach. We alsoextend it to simulate other parts of human and animal skin, and skin-tightclothes.

View record

This thesis describes a solid simulation method and its application to musculoskeletalsimulation. The presented solid simulation method features Eulerian discretizationand avoids mesh tangling during large deformation. Unlike existing Euleriansolid simulation methods, our method applies to elastoplastic material andvolume-preserving material. To further increase the utility of Eulerian simulationsfor solids, we introduce Lagrangian modes to the simulation and present a newsolver that handles close contact while simultaneously distributing motion betweenthe Lagrangian and Eulerian modes. This Eulerian-on-Lagrangian method enablesunbounded simulation domains and reduces the time step restrictions that oftenplague Eulerian simulation.We also introduce a framework for simulating the dynamics of musculoskeletalsystems, with volumetric muscles and a novel muscle activation model. Musclesare simulated using the solid simulator developed and therefore enjoys volumepreservation which is crucial for accurately capturing the dynamics of muscles andother biological tissues. Unlike previous work, in our system muscle deformationis tightly coupled to the dynamics of the skeletal system, and not added as an aftereffect. Our physiologically based muscle activation model utilizes knowledge ofthe active shapes of muscles, which can be manually drawn or easily obtained frommedical imaging data. Finally we demonstrate results with models derived fromMRI data and models designed for artistic effect.

View record

Eye movements have for a while provided us a closer view into how the brain commands the body. Particularly interesting are saccades: fast and accurate eye movements that allow us to scan our visual surroundings. One observation is that motor commands issued by the brain are corrupted by a signal-dependent noise. Moreover, the variance of the signal scales linearly with the control signal squared. It is assumed that such uncertainty in the dynamics introduces a probability distribution of the eye that the brain accounts for during motion planning.We propose a framework for computing the optimal control law for arbitrary dynamical systems, subject to noise, and where the cost function depends on a statistical distribution of the eye’s position. A key contribution of this framework is estimating the endpoint distribution of the plant using Monte Carlo sampling, which is done efficiently using commodity graphics hardware in parallel. We then describe a modified form of gradient descent for computing the optimal control law for an objective function prone to stochastic effects. We compare our approach to other methods, such as downhill simplex and Covariance-Matrix-Adaptation, which are considered “gradient-free” approaches to optimization. We finally conclude with several examples that show the framework successfully controlling saccades for different plant models of the oculomotor system: this includes a 3D torque-based model of the eye, and a a nonlinear model of the muscle actuator that drives the eye.

View record

We present the development of computational models of biological motor control used in two different types of eye movements --- gaze shifting and gaze stabilization. They are then implemented and tested on robotic systems. The thesis also investigates the application of these biological motor control strategies in robotics applications.We describe and test a non-linear control algorithm inspired by the behaviour of motor neurons in humans during extremely fast saccadic eye movements involved in gaze shifting. The algorithm is implemented on a robotic eye connected with a stiff camera cable, similar to the optic nerve. This adds a complicated non-linear stiffness to the plant. For high speed movement, our "pulse-step" controller operates in open-loop using an internal model of the eye plant learned from past measurements. We show that the controller approaches the performance seen in the human eye, producing fast movements with little overshoot. Interestingly, the controller reproduces the main sequence relationship observed in animal eye movements. We also model the control of eye movements that serve to stabilize its gaze direction. To test and evaluate this stabilization algorithm, we use a camera mounted on a robotic test platform that can have unknown perturbations in the horizontal plane. We show that using models of the vestibulo-ocular and optokinetic reflexes to control the camera allows the camera to be effectively stabilized using an inertial sensor and a single additional motor, without the need for a joint position sensor. The algorithm uses an adaptive controller based on a model of the vertebrate Cerebellum for velocity stabilization, with additional drift correction. A resolution-adaptive retinal slip algorithm that is robust to motion blur was also developed. We show that the resulting system can reduce camera image motion to about one pixel per frame on average even when the platform is rotated at 200 degrees per second. As a practical robotic application, we also demonstrate how the common task of face detection benefits from active gaze stabilization.

View record