Physically based soft-tissue simulations can create compelling digital creatures such as humans and animals. I will talk about our physically based simulation work at Ziva Dynamics, a startup company that I co-founded and jointly led as its CTO since 2015, now a part of Unity Technologies. We have created a soft-tissue FEM simulator that can model complex anatomically based creatures. We also invented a machine learning technology that can encode high-quality shapes into real-time deformers, to animate the shape of the character across its range of motion inside game engines and other interactive applications.

Working in an emerging field can be fun and incredibly rewarding, with the freedom to explore as we discover ways to achieve novel results. At the end of the development cycle, the process of moving those new ideas into production can often be overly burdened with constraints.

This talk looks at an implementation of machine learning in a visual effects pipeline. The workflow provides flexibility needed for continued development while also providing structure necessary for production. While covering the workflow from data acquisition to final composite, we will discuss how we built upon a solid vfx foundation, and how we used some simple design patterns allow for ongoing improvements to the techniques.

Measuring algorithmic bias is crucial both to assess algorithmic fairness, and to guide the improvement of algorithms. Current methods to measure algorithmic bias in computer vision, which are based on observational datasets, are inadequate for this task because they conflate algorithmic bias with dataset bias. To address this problem I will propose an experimental method for measuring algorithmic bias of face analysis algorithms, which manipulates directly the attributes of interest, e.g., gender and skin tone, in order to reveal causal links between attribute variation and performance change. The method is based on generating synthetic “transects” of matched sample images that are designed to differ along specific attributes while leaving other attributes constant. A crucial aspect of our approach is relying on the perception of human observers, both to guide manipulations, and to measure algorithmic bias. Besides allowing the measurement of algorithmic bias, synthetic transects have other advantages with respect to observational datasets: sampling attributes more evenly, allowing for reliable bias analysis on minority and intersectional groups, enabling prediction of bias in new scenarios, reducing ethical and legal challenges, making bias testing affordable and widely available. The method is validated by comparing it to a study that employs the traditional observational method for analyzing bias in gender classification algorithms. The two methods reach different conclusions. While the observational method reports gender and skin color biases, the experimental method reveals biases due to gender, hair length, age, and facial hair. (Joint work with Guha Balakrishnan)

 High-fidelity avatar creation for films and games is tied with complex capture equipment, massive data, a long production cycle, and intensive manual labour by a production team. And it may still be in the notorious Uncanny Valley. In this talk, we will explore how to produce a lifelike avatar in a low-cost way. We will show how to leverage deep learning networks to accelerate and simplify the industrial avatar production procedure from data capturing to animation. And bring photorealism to the next level!

At Netflix, we help storytellers share their stories with audiences all around the world through videos. Creating well cut videos (a.k.a. video editing) is a skill that mixes technology and art. In this talk, we'll present our journey into video understanding research at Netflix and how we've adapted this research field for building tools that enable our creative teams. Some of the topics we'll touch on include: multimodal learning, self supervised video representations, text-to-video clip retrieval, action recognition, image captioning, emotion recognition, and match cutting.