HOME - VLADLEN KOLTUNNEWSPROJECTSPUBLICATIONSLABTEACHINGCONTACT I am the Chief Scientist for Intelligent Systems at Intel. I lead an international lab of researchers working in machine learning, robotics, computer vision, computational science, and related areas. LAB - VLADLEN KOLTUN Jack Wang, postdoc, 2010-2013 Next position: Assistant Professor at the University of Hong Kong EXPLORING SELF-ATTENTION FOR IMAGE RECOGNITION Exploring Self-attention for Image Recognition Hengshuang Zhao CUHK Jiaya Jia CUHK Vladlen Koltun Intel Labs Abstract Recent work has shown that self-attention can serve as DYNAMIC LOW-LIGHT IMAGING WITH QUANTA IMAGE SENSORS Dynamic Low-light Imaging with Quanta Image Sensors Yiheng Chi 1, Abhiram Gnanasambandam , Vladlen Koltun2, and Stanley H. Chan1 1 School of ECE, Purdue University, West Lafayette, IN 47907, USA 2 Intel Labs, Santa Clara, CA 95054, USA Abstract. Imaging FAST GLOBAL REGISTRATION Fast Global Registration 5 µ=1 µ=0.25 µ=4 µ=16 (a) Geman-McClure penalty (b) Objective function Fig.2. Illustration of graduated non-convexity. As decreases, the objective function for the NERF++: ANALYZING AND IMPROVING NEURAL RADIANCE FIELDS Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings, including 360-degree capture of bounded scenes and forward MSEG: A COMPOSITE DATASET FOR MULTI-DOMAIN SEMANTIC MSeg: A Composite Dataset for Multi-domain Semantic Segmentation John Lambert 1,3, Zhuang Liu 1,2, Ozan Sener1, James Hays3,4, and Vladlen Koltun1 1Intel Labs, 2University of California, Berkeley, 3Georgia Institute of Technology, 4Argo AI Input image Ground truth ADE20K model Mapillary model COCO model MSeg model ACCURATE OPTICAL FLOW VIA DIRECT COST VOLUME PROCESSING Accurate Optical Flow via Direct Cost Volume Processing Jia Xu Rene Ranftl Vladlen Koltun´ Intel Labs Abstract We present an optical flow estimation approach that op- SPEECH DENOISING WITH DEEP FEATURE LOSSES Speech Denoising With Deep Feature Losses Franc¸ois G. Germain1, Qifeng Chen2, and Vladlen Koltun3 1CCRMA, Stanford University, Stanford, CA, USA 2Department of Computer Science and Engineering, HKUST, Kowloon, Hong Kong 3Intelligent Systems Lab, Intel Labs, Santa Clara, CA, USA francois@ccrma.stanford.edu, cqf@ust.hk, vladlen.koltun@intel.com Abstract COLORED POINT CLOUD REGISTRATION REVISITED SUPPLEMENTARY Colored Point Cloud Registration Revisited Supplementary Material Jaesik Park Qian-Yi Zhou Vladlen Koltun Intel Labs A. RGB-D Image Alignment Section 3 introduced a joint photometric and geomet- HOME - VLADLEN KOLTUNNEWSPROJECTSPUBLICATIONSLABTEACHINGCONTACT I am the Chief Scientist for Intelligent Systems at Intel. I lead an international lab of researchers working in machine learning, robotics, computer vision, computational science, and related areas. LAB - VLADLEN KOLTUN Jack Wang, postdoc, 2010-2013 Next position: Assistant Professor at the University of Hong Kong EXPLORING SELF-ATTENTION FOR IMAGE RECOGNITION Exploring Self-attention for Image Recognition Hengshuang Zhao CUHK Jiaya Jia CUHK Vladlen Koltun Intel Labs Abstract Recent work has shown that self-attention can serve as DYNAMIC LOW-LIGHT IMAGING WITH QUANTA IMAGE SENSORS Dynamic Low-light Imaging with Quanta Image Sensors Yiheng Chi 1, Abhiram Gnanasambandam , Vladlen Koltun2, and Stanley H. Chan1 1 School of ECE, Purdue University, West Lafayette, IN 47907, USA 2 Intel Labs, Santa Clara, CA 95054, USA Abstract. Imaging FAST GLOBAL REGISTRATION Fast Global Registration 5 µ=1 µ=0.25 µ=4 µ=16 (a) Geman-McClure penalty (b) Objective function Fig.2. Illustration of graduated non-convexity. As decreases, the objective function for the NERF++: ANALYZING AND IMPROVING NEURAL RADIANCE FIELDS Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings, including 360-degree capture of bounded scenes and forward MSEG: A COMPOSITE DATASET FOR MULTI-DOMAIN SEMANTIC MSeg: A Composite Dataset for Multi-domain Semantic Segmentation John Lambert 1,3, Zhuang Liu 1,2, Ozan Sener1, James Hays3,4, and Vladlen Koltun1 1Intel Labs, 2University of California, Berkeley, 3Georgia Institute of Technology, 4Argo AI Input image Ground truth ADE20K model Mapillary model COCO model MSeg model ACCURATE OPTICAL FLOW VIA DIRECT COST VOLUME PROCESSING Accurate Optical Flow via Direct Cost Volume Processing Jia Xu Rene Ranftl Vladlen Koltun´ Intel Labs Abstract We present an optical flow estimation approach that op- SPEECH DENOISING WITH DEEP FEATURE LOSSES Speech Denoising With Deep Feature Losses Franc¸ois G. Germain1, Qifeng Chen2, and Vladlen Koltun3 1CCRMA, Stanford University, Stanford, CA, USA 2Department of Computer Science and Engineering, HKUST, Kowloon, Hong Kong 3Intelligent Systems Lab, Intel Labs, Santa Clara, CA, USA francois@ccrma.stanford.edu, cqf@ust.hk, vladlen.koltun@intel.com Abstract COLORED POINT CLOUD REGISTRATION REVISITED SUPPLEMENTARY Colored Point Cloud Registration Revisited Supplementary Material Jaesik Park Qian-Yi Zhou Vladlen Koltun Intel Labs A. RGB-D Image Alignment Section 3 introduced a joint photometric and geomet- NEWS - VLADLEN KOLTUN Adobe released a product based on our data-driven 3D modeling research at Stanford. The original research was described in papers published at SIGGRAPH Asia 2010 and SIGGRAPH 2011, and was the subject of Siddhartha Chaudhuri’s PhD thesis. PROJECTS - VLADLEN KOLTUN Machine Learning. We are working on core models and algorithms in machine learning. One long-term interest is in deep network architectures. We are interested both in new layers and operators, and in new macroscopic structures and connectivity patterns.

PUBLICATIONS

Current robots are either expensive or make significant compromises on sensory richness, computational power, and communication capabilities. We propose to leverage smartphones to equip robots with extensive sensor suites, powerful computational abilities, state-of-the-art communication channels, and access to a thriving software ecosystem.

VLADLEN KOLTUN

Vladlen Koltun Curriculum Vitae vladlen.info vladlen.koltun@intel.com Born Nov 1980 EDUCATION 2002 Ph.D., Computer Science, Tel Aviv University, with distinction TANKS AND TEMPLES: BENCHMARKING LARGE-SCALE SCENE We present a benchmark for image-based 3D reconstruction. The benchmark sequences were acquired outside the lab, in realistic conditions. Ground-truth data was captured using an industrial laser

scanner.

HOME - VLADLEN KOLTUNTRANSLATE THIS PAGE Home - Vladlen Koltun LEARNING COMPACT GEOMETRIC FEATURES S n p rmin Figure 2. Our input parameterization, illustrated in two dimensions for clarity. The sphere S is centered at the point p. In this two-dimensional illustration, the interior of S is subdivided

into three

WHAT DO SINGLE-VIEW 3D RECONSTRUCTION NETWORKS LEARN? What Do Single-view 3D Reconstruction Networks Learn? Maxim Tatarchenko 1, Stephan R. Richter 2, Rene Ranftl´ 2, Zhuwen Li2, Vladlen Koltun2, and Thomas Brox1 1University of Freiburg 2Intel Labs Figure 1. We provide evidence that state-of-the-art single-view 3D reconstruction methods (AtlasNet (light green, 0:38 IoU) , OGN AN EMPIRICAL EVALUATION OF GENERIC CONVOLUTIONAL AND An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling To our knowledge, the presented study is the most extensive EFFICIENT INFERENCE IN FULLY CONNECTED CRFS WITH GAUSSIAN E U˘Qrefers to the expected value under the distribution Q.We use the fact that the Shanon en-tropy E U˘Q = P i E U i˘Q i decomposes when Q(X) = Q i Q i(X i), due to linearity of expectation. The marginal Q i(x i) that minimizes the KL-divergence is found by analytically minimizing a La- grangian that consists of all terms in D(QkP) plus Lagrange multipliers assuring HOME - VLADLEN KOLTUNNEWSPROJECTSPUBLICATIONSLABTEACHINGCONTACT Home - Vladlen Koltun. I am the Chief Scientist for Intelligent Systems at Intel. I lead an international lab of researchers working in machine learning, robotics, computer vision, computational science, and related areas. Our researchers are based on four continents, with extensive collaborations irrespective of geographic location. DYNAMIC LOW-LIGHT IMAGING WITH QUANTA IMAGE SENSORS Dynamic Low-light Imaging with Quanta Image Sensors Yiheng Chi 1, Abhiram Gnanasambandam , Vladlen Koltun2, and Stanley H. Chan1 1 School of ECE, Purdue University, West Lafayette, IN 47907, USA 2 Intel Labs, Santa Clara, CA 95054, USA Abstract. Imaging LAB - VLADLEN KOLTUN Jack Wang, postdoc, 2010-2013 Next position: Assistant Professor at the University of Hong Kong FAST GLOBAL REGISTRATION Fast Global Registration 5 µ=1 µ=0.25 µ=4 µ=16 (a) Geman-McClure penalty (b) Objective function Fig.2. Illustration of graduated non-convexity. As decreases, the objective function for the SPEECH DENOISING WITH DEEP FEATURE LOSSES Speech Denoising With Deep Feature Losses Franc¸ois G. Germain1, Qifeng Chen2, and Vladlen Koltun3 1CCRMA, Stanford University, Stanford, CA, USA 2Department of Computer Science and Engineering, HKUST, Kowloon, Hong Kong 3Intelligent Systems Lab, Intel Labs, Santa Clara, CA, USA francois@ccrma.stanford.edu, cqf@ust.hk, vladlen.koltun@intel.com Abstract NERF++: ANALYZING AND IMPROVING NEURAL RADIANCE FIELDS Abstract. Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings, including 360-degree capture of bounded scenes and forward-facing capture of bounded and unbounded scenes. NeRF fits multi-layer perceptrons (MLPs) representing view-invariant opacity and view-dependent color volumes

to a set of

MSEG: A COMPOSITE DATASET FOR MULTI-DOMAIN SEMANTIC MSeg: A Composite Dataset for Multi-domain Semantic Segmentation John Lambert 1,3, Zhuang Liu 1,2, Ozan Sener1, James Hays3,4, and Vladlen Koltun1 1Intel Labs, 2University of California, Berkeley, 3Georgia Institute of Technology, 4Argo AI Input image Ground truth ADE20K model Mapillary model COCO model MSeg model EXPLORING SELF-ATTENTION FOR IMAGE RECOGNITION 3.1. Pairwise Self-attention We explore two types of self-attention. The first, which we refer to as pairwise, has the following form: y i

= X j2R(i)

ACCURATE OPTICAL FLOW VIA DIRECT COST VOLUME PROCESSING Accurate Optical Flow via Direct Cost Volume Processing Jia Xu Rene Ranftl Vladlen Koltun´ Intel Labs Abstract We present an optical flow estimation approach that op- COLORED POINT CLOUD REGISTRATION REVISITED SUPPLEMENTARY Colored Point Cloud Registration Revisited Supplementary Material Jaesik Park Qian-Yi Zhou Vladlen Koltun Intel Labs A. RGB-D Image Alignment Section 3 introduced a joint photometric and geomet- HOME - VLADLEN KOLTUNNEWSPROJECTSPUBLICATIONSLABTEACHINGCONTACT Home - Vladlen Koltun. I am the Chief Scientist for Intelligent Systems at Intel. I lead an international lab of researchers working in machine learning, robotics, computer vision, computational science, and related areas. Our researchers are based on four continents, with extensive collaborations irrespective of geographic location. DYNAMIC LOW-LIGHT IMAGING WITH QUANTA IMAGE SENSORS Dynamic Low-light Imaging with Quanta Image Sensors Yiheng Chi 1, Abhiram Gnanasambandam , Vladlen Koltun2, and Stanley H. Chan1 1 School of ECE, Purdue University, West Lafayette, IN 47907, USA 2 Intel Labs, Santa Clara, CA 95054, USA Abstract. Imaging LAB - VLADLEN KOLTUN Jack Wang, postdoc, 2010-2013 Next position: Assistant Professor at the University of Hong Kong FAST GLOBAL REGISTRATION Fast Global Registration 5 µ=1 µ=0.25 µ=4 µ=16 (a) Geman-McClure penalty (b) Objective function Fig.2. Illustration of graduated non-convexity. As decreases, the objective function for the SPEECH DENOISING WITH DEEP FEATURE LOSSES Speech Denoising With Deep Feature Losses Franc¸ois G. Germain1, Qifeng Chen2, and Vladlen Koltun3 1CCRMA, Stanford University, Stanford, CA, USA 2Department of Computer Science and Engineering, HKUST, Kowloon, Hong Kong 3Intelligent Systems Lab, Intel Labs, Santa Clara, CA, USA francois@ccrma.stanford.edu, cqf@ust.hk, vladlen.koltun@intel.com Abstract NERF++: ANALYZING AND IMPROVING NEURAL RADIANCE FIELDS Abstract. Neural Radiance Fields (NeRF) achieve impressive view synthesis results for a variety of capture settings, including 360-degree capture of bounded scenes and forward-facing capture of bounded and unbounded scenes. NeRF fits multi-layer perceptrons (MLPs) representing view-invariant opacity and view-dependent color volumes

to a set of

MSEG: A COMPOSITE DATASET FOR MULTI-DOMAIN SEMANTIC MSeg: A Composite Dataset for Multi-domain Semantic Segmentation John Lambert 1,3, Zhuang Liu 1,2, Ozan Sener1, James Hays3,4, and Vladlen Koltun1 1Intel Labs, 2University of California, Berkeley, 3Georgia Institute of Technology, 4Argo AI Input image Ground truth ADE20K model Mapillary model COCO model MSeg model EXPLORING SELF-ATTENTION FOR IMAGE RECOGNITION 3.1. Pairwise Self-attention We explore two types of self-attention. The first, which we refer to as pairwise, has the following form: y i

= X j2R(i)

ACCURATE OPTICAL FLOW VIA DIRECT COST VOLUME PROCESSING Accurate Optical Flow via Direct Cost Volume Processing Jia Xu Rene Ranftl Vladlen Koltun´ Intel Labs Abstract We present an optical flow estimation approach that op- COLORED POINT CLOUD REGISTRATION REVISITED SUPPLEMENTARY Colored Point Cloud Registration Revisited Supplementary Material Jaesik Park Qian-Yi Zhou Vladlen Koltun Intel Labs A. RGB-D Image Alignment Section 3 introduced a joint photometric and geomet-

PUBLICATIONS

Drinking from a Firehose: Continual Learning with Web-scale Natural Language. Hexiang Hu, Ozan Sener, Fei Sha, and Vladlen Koltun. Pre-print, 2007.09335, 2020. Continual learning systems will interact with humans, with each other, and with the physical world through time

—

PROJECTS - VLADLEN KOLTUN Sensorimotor Control and Simulation. We are working on sensorimotor control: learning to act based on raw sensory input. Our models and algorithms aim to support flexible operation in complex and dynamic three-dimensional environments. We are inspired by applications such as autonomous driving and household robotics, and by scientific

curiosity.

NEWS - VLADLEN KOLTUN Posted on March 4, 2018. Four papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR). Semi-parametric Image Synthesis was selected for full oral presentation at the conference (2.1% acceptance rate). Tangent Convolutions for Dense Prediction in 3D was selected for a spotlight oral.

VLADLEN KOLTUN

Vladlen Koltun Curriculum Vitae vladlen.info vladlen.koltun@intel.com Born Nov 1980 EDUCATION 2002 Ph.D., Computer Science, Tel Aviv University, with distinction PAPER ACCEPTED TO ICLR 2021 Paper accepted to ICLR 2021. Posted January 14, 2021. The paper Large Batch Simulation for Deep Reinforcement Learning was accepted to the International Conference on Learning Representations (ICLR). ← Paper published in Science Robotics and featured on the cover. HOME - VLADLEN KOLTUNTRANSLATE THIS PAGE Home - Vladlen Koltun LEARNING COMPACT GEOMETRIC FEATURES S n p rmin Figure 2. Our input parameterization, illustrated in two dimensions for clarity. The sphere S is centered at the point p. In this two-dimensional illustration, the interior of S is subdivided

into three

END-TO-END DRIVING VIA CONDITIONAL IMITATION LEARNING End-to-end Driving via Conditional Imitation Learning Felipe Codevilla 1;2 Matthias Muller¨ 3 Antonio Lopez´ 2 Vladlen Koltun 1Alexey Dosovitskiy (a) Aerial view of test environment (b) Vision-based driving, view from onboard camera (c) Side view of vehicle WHAT DO SINGLE-VIEW 3D RECONSTRUCTION NETWORKS LEARN? What Do Single-view 3D Reconstruction Networks Learn? Maxim Tatarchenko 1, Stephan R. Richter 2, Rene Ranftl´ 2, Zhuwen Li2, Vladlen Koltun2, and Thomas Brox1 1University of Freiburg 2Intel Labs Figure 1. We provide evidence that state-of-the-art single-view 3D reconstruction methods (AtlasNet (light green, 0:38 IoU) , OGN AN EMPIRICAL EVALUATION OF GENERIC CONVOLUTIONAL AND An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling To our knowledge, the presented study is the most extensive HOME - VLADLEN KOLTUNNEWSPROJECTSPUBLICATIONSLABTEACHINGCONTACT Home - Vladlen Koltun. I am the Chief Scientist for Intelligent Systems at Intel. I lead an international lab of researchers working in machine learning, robotics, computer vision, computational science, and related areas. Our researchers are based on four continents, with extensive collaborations irrespective of geographic location. LAB - VLADLEN KOLTUN Jack Wang, postdoc, 2010-2013 Next position: Assistant Professor at the University of Hong Kong DYNAMIC LOW-LIGHT IMAGING WITH QUANTA IMAGE SENSORS Dynamic Low-light Imaging with Quanta Image Sensors Yiheng Chi 1, Abhiram Gnanasambandam , Vladlen Koltun2, and Stanley H. Chan1 1 School of ECE, Purdue University, West Lafayette, IN 47907, USA 2 Intel Labs, Santa Clara, CA 95054, USA Abstract. Imaging FAST GLOBAL REGISTRATION Abstract. We present an algorithm for fast global registration of partially overlapping 3D surfaces. The algorithm operates on candidate matches that cover the surfaces. A single objective is optimized to align the surfaces and disable false matches. The objective is defined densely over the surfaces and the optimization achieves tight COLOR MAP OPTIMIZATION FOR 3D RECONSTRUCTION WITH CONSUMER Abstract. We present a global optimization approach for mapping color images onto geometric reconstructions. Range and color videos produced by consumer-grade RGB-D cameras suffer from noise and optical distortions, which impede accurate mapping of the acquired color data to the reconstructed geometry. Our approach addresses these sources of EXPLORING SELF-ATTENTION FOR IMAGE RECOGNITION 3.1. Pairwise Self-attention We explore two types of self-attention. The first, which we refer to as pairwise, has the following form: y i

= X j2R(i)

MSEG: A COMPOSITE DATASET FOR MULTI-DOMAIN SEMANTIC MSeg: A Composite Dataset for Multi-domain Semantic Segmentation John Lambert 1,3, Zhuang Liu 1,2, Ozan Sener1, James Hays3,4, and Vladlen Koltun1 1Intel Labs, 2University of California, Berkeley, 3Georgia Institute of Technology, 4Argo AI Input image Ground truth ADE20K model Mapillary model COCO model MSeg model ACCURATE OPTICAL FLOW VIA DIRECT COST VOLUME PROCESSINGDENSE OPTICAL FLOWDETERMINING OPTICAL FLOWOPTICAL FLOW APPLICATIONOPTICAL FLOW

CAMERA

Accurate Optical Flow via Direct Cost Volume Processing Jia Xu Rene Ranftl Vladlen Koltun´ Intel Labs Abstract We present an optical flow estimation approach that op- SPEECH DENOISING WITH DEEP FEATURE LOSSES Speech Denoising With Deep Feature Losses Franc¸ois G. Germain1, Qifeng Chen2, and Vladlen Koltun3 1CCRMA, Stanford University, Stanford, CA, USA 2Department of Computer Science and Engineering, HKUST, Kowloon, Hong Kong 3Intelligent Systems Lab, Intel Labs, Santa Clara, CA, USA francois@ccrma.stanford.edu, cqf@ust.hk, vladlen.koltun@intel.com Abstract HOME - VLADLEN KOLTUN Home - Vladlen Koltun HOME - VLADLEN KOLTUNNEWSPROJECTSPUBLICATIONSLABTEACHINGCONTACT Home - Vladlen Koltun. I am the Chief Scientist for Intelligent Systems at Intel. I lead an international lab of researchers working in machine learning, robotics, computer vision, computational science, and related areas. Our researchers are based on four continents, with extensive collaborations irrespective of geographic location. LAB - VLADLEN KOLTUN Jack Wang, postdoc, 2010-2013 Next position: Assistant Professor at the University of Hong Kong DYNAMIC LOW-LIGHT IMAGING WITH QUANTA IMAGE SENSORS Dynamic Low-light Imaging with Quanta Image Sensors Yiheng Chi 1, Abhiram Gnanasambandam , Vladlen Koltun2, and Stanley H. Chan1 1 School of ECE, Purdue University, West Lafayette, IN 47907, USA 2 Intel Labs, Santa Clara, CA 95054, USA Abstract. Imaging FAST GLOBAL REGISTRATION Abstract. We present an algorithm for fast global registration of partially overlapping 3D surfaces. The algorithm operates on candidate matches that cover the surfaces. A single objective is optimized to align the surfaces and disable false matches. The objective is defined densely over the surfaces and the optimization achieves tight COLOR MAP OPTIMIZATION FOR 3D RECONSTRUCTION WITH CONSUMER Abstract. We present a global optimization approach for mapping color images onto geometric reconstructions. Range and color videos produced by consumer-grade RGB-D cameras suffer from noise and optical distortions, which impede accurate mapping of the acquired color data to the reconstructed geometry. Our approach addresses these sources of EXPLORING SELF-ATTENTION FOR IMAGE RECOGNITION 3.1. Pairwise Self-attention We explore two types of self-attention. The first, which we refer to as pairwise, has the following form: y i

= X j2R(i)

MSEG: A COMPOSITE DATASET FOR MULTI-DOMAIN SEMANTIC MSeg: A Composite Dataset for Multi-domain Semantic Segmentation John Lambert 1,3, Zhuang Liu 1,2, Ozan Sener1, James Hays3,4, and Vladlen Koltun1 1Intel Labs, 2University of California, Berkeley, 3Georgia Institute of Technology, 4Argo AI Input image Ground truth ADE20K model Mapillary model COCO model MSeg model ACCURATE OPTICAL FLOW VIA DIRECT COST VOLUME PROCESSINGDENSE OPTICAL FLOWDETERMINING OPTICAL FLOWOPTICAL FLOW APPLICATIONOPTICAL FLOW

CAMERA

Accurate Optical Flow via Direct Cost Volume Processing Jia Xu Rene Ranftl Vladlen Koltun´ Intel Labs Abstract We present an optical flow estimation approach that op- SPEECH DENOISING WITH DEEP FEATURE LOSSES Speech Denoising With Deep Feature Losses Franc¸ois G. Germain1, Qifeng Chen2, and Vladlen Koltun3 1CCRMA, Stanford University, Stanford, CA, USA 2Department of Computer Science and Engineering, HKUST, Kowloon, Hong Kong 3Intelligent Systems Lab, Intel Labs, Santa Clara, CA, USA francois@ccrma.stanford.edu, cqf@ust.hk, vladlen.koltun@intel.com Abstract HOME - VLADLEN KOLTUN Home - Vladlen Koltun PROJECTS - VLADLEN KOLTUN Machine Learning. We are working on core models and algorithms in machine learning. One long-term interest is in deep network architectures. We are interested both in new layers and operators, and in new macroscopic structures and connectivity patterns.

PUBLICATIONS

Drinking from a Firehose: Continual Learning with Web-scale Natural Language. Hexiang Hu, Ozan Sener, Fei Sha, and Vladlen Koltun. Pre-print, 2007.09335, 2020. Continual learning systems will interact with humans, with each other, and with the physical world through time

—

LEARNING TO SEE IN THE DARK Abstract. Imaging in low light is challenging due to low photon count and low SNR. Short-exposure images suffer from noise, while long exposure can induce blur and is often impractical. A variety of denoising, deblurring, and enhancement techniques have been proposed, but their effectiveness is limited in extreme conditions, such as

video-rate

EXPLORING SELF-ATTENTION FOR IMAGE RECOGNITION We explore variations of self-attention and assess their effectiveness for image recognition. We consider two forms of self-attention. One is pairwise self-attention, which generalizes standard dot-product attention and is fundamentally a set operator. The other is patchwise self-attention, which is strictly more powerful than convolution. GEODESIC OBJECT PROPOSALS This set of candidates can be used for object recognition, segmentation, and other object-based image parsing tasks. To generate the proposals, we identify critical level sets in geodesic distance transforms computed for seeds placed in the image. The seeds are placed by specially trained classifiers that are optimized to discover

objects.

BEAUTY AND THE BEAST: OPTIMAL METHODS MEET LEARNING FOR Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing Elia Kaufmann 1, Mathias Gehrig , Philipp Foehn , Ren´e Ranftl 2, Alexey Dosovitskiy , Vladlen Koltun2, Davide Scaramuzza1 Abstract—Autonomous micro aerial vehicles still struggle TANKS AND TEMPLES: BENCHMARKING LARGE-SCALE SCENE The benchmark includes both outdoor scenes and indoor environments. High-resolution video sequences are provided as input, supporting the development of novel pipelines that take advantage of video input to increase reconstruction fidelity. We report the performance of many image-based 3D reconstruction pipelines on the new benchmark. FAST GLOBAL REGISTRATION Fast Global Registration 5 µ=1 µ=0.25 µ=4 µ=16 (a) Geman-McClure penalty (b) Objective function Fig.2. Illustration of graduated non-convexity. As decreases, the objective function for the HOME - VLADLEN KOLTUN Home - Vladlen Koltun DOES COMPUTER VISION MATTER FOR ACTION? Focus Intel and UT Austin1 Does Computer Vision Matter for Action? BRADY ZHOU1*,PHILIPP KRÄHENBÜHL1,2, AND VLADLEN KOLTUN1 1Intel Labs 2University of Texas at Austin *Corresponding author: brady.zhou@utexas.edu Controlled experiments indicate that explicit intermediate representations help action. HOME - VLADLEN KOLTUNNEWSPROJECTSPUBLICATIONSLABTEACHINGCONTACT Home - Vladlen Koltun. I am the Chief Scientist for Intelligent Systems at Intel. I lead an international lab of researchers working in machine learning, robotics, computer vision, computational science, and related areas. Our researchers are based on four continents, with extensive collaborations irrespective of geographic location. LAB - VLADLEN KOLTUN Jack Wang, postdoc, 2010-2013 Next position: Assistant Professor at the University of Hong Kong DYNAMIC LOW-LIGHT IMAGING WITH QUANTA IMAGE SENSORS Dynamic Low-light Imaging with Quanta Image Sensors Yiheng Chi 1, Abhiram Gnanasambandam , Vladlen Koltun2, and Stanley H. Chan1 1 School of ECE, Purdue University, West Lafayette, IN 47907, USA 2 Intel Labs, Santa Clara, CA 95054, USA Abstract. Imaging FAST GLOBAL REGISTRATION Abstract. We present an algorithm for fast global registration of partially overlapping 3D surfaces. The algorithm operates on candidate matches that cover the surfaces. A single objective is optimized to align the surfaces and disable false matches. The objective is defined densely over the surfaces and the optimization achieves tight COLOR MAP OPTIMIZATION FOR 3D RECONSTRUCTION WITH CONSUMER Abstract. We present a global optimization approach for mapping color images onto geometric reconstructions. Range and color videos produced by consumer-grade RGB-D cameras suffer from noise and optical distortions, which impede accurate mapping of the acquired color data to the reconstructed geometry. Our approach addresses these sources of EXPLORING SELF-ATTENTION FOR IMAGE RECOGNITION 3.1. Pairwise Self-attention We explore two types of self-attention. The first, which we refer to as pairwise, has the following form: y i

= X j2R(i)

MSEG: A COMPOSITE DATASET FOR MULTI-DOMAIN SEMANTIC MSeg: A Composite Dataset for Multi-domain Semantic Segmentation John Lambert 1,3, Zhuang Liu 1,2, Ozan Sener1, James Hays3,4, and Vladlen Koltun1 1Intel Labs, 2University of California, Berkeley, 3Georgia Institute of Technology, 4Argo AI Input image Ground truth ADE20K model Mapillary model COCO model MSeg model ACCURATE OPTICAL FLOW VIA DIRECT COST VOLUME PROCESSINGDENSE OPTICAL FLOWDETERMINING OPTICAL FLOWOPTICAL FLOW APPLICATIONOPTICAL FLOW

CAMERA

Accurate Optical Flow via Direct Cost Volume Processing Jia Xu Rene Ranftl Vladlen Koltun´ Intel Labs Abstract We present an optical flow estimation approach that op- SPEECH DENOISING WITH DEEP FEATURE LOSSES Speech Denoising With Deep Feature Losses Franc¸ois G. Germain1, Qifeng Chen2, and Vladlen Koltun3 1CCRMA, Stanford University, Stanford, CA, USA 2Department of Computer Science and Engineering, HKUST, Kowloon, Hong Kong 3Intelligent Systems Lab, Intel Labs, Santa Clara, CA, USA francois@ccrma.stanford.edu, cqf@ust.hk, vladlen.koltun@intel.com Abstract HOME - VLADLEN KOLTUN Home - Vladlen Koltun HOME - VLADLEN KOLTUNNEWSPROJECTSPUBLICATIONSLABTEACHINGCONTACT Home - Vladlen Koltun. I am the Chief Scientist for Intelligent Systems at Intel. I lead an international lab of researchers working in machine learning, robotics, computer vision, computational science, and related areas. Our researchers are based on four continents, with extensive collaborations irrespective of geographic location. LAB - VLADLEN KOLTUN Jack Wang, postdoc, 2010-2013 Next position: Assistant Professor at the University of Hong Kong DYNAMIC LOW-LIGHT IMAGING WITH QUANTA IMAGE SENSORS Dynamic Low-light Imaging with Quanta Image Sensors Yiheng Chi 1, Abhiram Gnanasambandam , Vladlen Koltun2, and Stanley H. Chan1 1 School of ECE, Purdue University, West Lafayette, IN 47907, USA 2 Intel Labs, Santa Clara, CA 95054, USA Abstract. Imaging FAST GLOBAL REGISTRATION Abstract. We present an algorithm for fast global registration of partially overlapping 3D surfaces. The algorithm operates on candidate matches that cover the surfaces. A single objective is optimized to align the surfaces and disable false matches. The objective is defined densely over the surfaces and the optimization achieves tight COLOR MAP OPTIMIZATION FOR 3D RECONSTRUCTION WITH CONSUMER Abstract. We present a global optimization approach for mapping color images onto geometric reconstructions. Range and color videos produced by consumer-grade RGB-D cameras suffer from noise and optical distortions, which impede accurate mapping of the acquired color data to the reconstructed geometry. Our approach addresses these sources of EXPLORING SELF-ATTENTION FOR IMAGE RECOGNITION 3.1. Pairwise Self-attention We explore two types of self-attention. The first, which we refer to as pairwise, has the following form: y i

= X j2R(i)

MSEG: A COMPOSITE DATASET FOR MULTI-DOMAIN SEMANTIC MSeg: A Composite Dataset for Multi-domain Semantic Segmentation John Lambert 1,3, Zhuang Liu 1,2, Ozan Sener1, James Hays3,4, and Vladlen Koltun1 1Intel Labs, 2University of California, Berkeley, 3Georgia Institute of Technology, 4Argo AI Input image Ground truth ADE20K model Mapillary model COCO model MSeg model ACCURATE OPTICAL FLOW VIA DIRECT COST VOLUME PROCESSINGDENSE OPTICAL FLOWDETERMINING OPTICAL FLOWOPTICAL FLOW APPLICATIONOPTICAL FLOW

CAMERA

Accurate Optical Flow via Direct Cost Volume Processing Jia Xu Rene Ranftl Vladlen Koltun´ Intel Labs Abstract We present an optical flow estimation approach that op- SPEECH DENOISING WITH DEEP FEATURE LOSSES Speech Denoising With Deep Feature Losses Franc¸ois G. Germain1, Qifeng Chen2, and Vladlen Koltun3 1CCRMA, Stanford University, Stanford, CA, USA 2Department of Computer Science and Engineering, HKUST, Kowloon, Hong Kong 3Intelligent Systems Lab, Intel Labs, Santa Clara, CA, USA francois@ccrma.stanford.edu, cqf@ust.hk, vladlen.koltun@intel.com Abstract HOME - VLADLEN KOLTUN Home - Vladlen Koltun PROJECTS - VLADLEN KOLTUN Machine Learning. We are working on core models and algorithms in machine learning. One long-term interest is in deep network architectures. We are interested both in new layers and operators, and in new macroscopic structures and connectivity patterns.

PUBLICATIONS

Drinking from a Firehose: Continual Learning with Web-scale Natural Language. Hexiang Hu, Ozan Sener, Fei Sha, and Vladlen Koltun. Pre-print, 2007.09335, 2020. Continual learning systems will interact with humans, with each other, and with the physical world through time

—

LEARNING TO SEE IN THE DARK Abstract. Imaging in low light is challenging due to low photon count and low SNR. Short-exposure images suffer from noise, while long exposure can induce blur and is often impractical. A variety of denoising, deblurring, and enhancement techniques have been proposed, but their effectiveness is limited in extreme conditions, such as

video-rate

EXPLORING SELF-ATTENTION FOR IMAGE RECOGNITION We explore variations of self-attention and assess their effectiveness for image recognition. We consider two forms of self-attention. One is pairwise self-attention, which generalizes standard dot-product attention and is fundamentally a set operator. The other is patchwise self-attention, which is strictly more powerful than convolution. GEODESIC OBJECT PROPOSALS This set of candidates can be used for object recognition, segmentation, and other object-based image parsing tasks. To generate the proposals, we identify critical level sets in geodesic distance transforms computed for seeds placed in the image. The seeds are placed by specially trained classifiers that are optimized to discover

objects.

BEAUTY AND THE BEAST: OPTIMAL METHODS MEET LEARNING FOR Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing Elia Kaufmann 1, Mathias Gehrig , Philipp Foehn , Ren´e Ranftl 2, Alexey Dosovitskiy , Vladlen Koltun2, Davide Scaramuzza1 Abstract—Autonomous micro aerial vehicles still struggle TANKS AND TEMPLES: BENCHMARKING LARGE-SCALE SCENE The benchmark includes both outdoor scenes and indoor environments. High-resolution video sequences are provided as input, supporting the development of novel pipelines that take advantage of video input to increase reconstruction fidelity. We report the performance of many image-based 3D reconstruction pipelines on the new benchmark. FAST GLOBAL REGISTRATION Fast Global Registration 5 µ=1 µ=0.25 µ=4 µ=16 (a) Geman-McClure penalty (b) Objective function Fig.2. Illustration of graduated non-convexity. As decreases, the objective function for the HOME - VLADLEN KOLTUN Home - Vladlen Koltun DOES COMPUTER VISION MATTER FOR ACTION? Focus Intel and UT Austin1 Does Computer Vision Matter for Action? BRADY ZHOU1*,PHILIPP KRÄHENBÜHL1,2, AND VLADLEN KOLTUN1 1Intel Labs 2University of Texas at Austin *Corresponding author: brady.zhou@utexas.edu Controlled experiments indicate that explicit intermediate representations help action.

* Home

* /

* News

* /

* Projects

* /

* Publications

* /

* Lab

* /

* Teaching

* /

* Contact

* /

I am the Chief Scientist for Intelligent Systems at Intel. I lead an international lab of researchers working in machine learning, robotics, computer vision, computational science, and related areas. Our researchers are based on four continents, with extensive collaborations irrespective of geographic location. WE ARE HIRING research scientists, research engineers, postdocs, pre-docs, and interns. Email me with your CV if interested.

CV / Publications

/ Google Scholar

/ Facebook

Selected Papers

A few personal favorites. * Deep network architectures * Deep Equilibrium Models

, NeurIPS

2019

* Trellis Networks for Sequence Modeling

,

ICLR 2019

* Multi-Scale Context Aggregation by Dilated Convolutions

,

ICLR 2016

* Machine learning, algorithms * Deep Fundamental Matrix Estimation

,

ECCV 2018

* Robust Continuous Clustering

, PNAS

2017

* Efficient Inference in Fully Connected CRFs with Gaussian Edge

Potentials

,

NIPS 2011

* Robotics

* Learning Quadrupedal Locomotion over Challenging Terrain

,

Science Robotics 2020 * Sensorimotor control * Learning by Cheating

, CoRL 2019

* Does Computer Vision Matter for Action?

,

Science Robotics 2019 * Learning to Act by Predicting the Future

,

ICLR 2017

* 3D

* Fast Global Registration

, ECCV

2016

* Robust Reconstruction of Indoor Scenes

,

CVPR 2015

* Simulation

* Playing for Benchmarks

, ICCV 2017

* Playing for Data: Ground Truth from Computer Games

,

ECCV 2016

* Image processing and synthesis * Photographic Image Synthesis with Cascaded Refinement Networks

,

ICCV 2017

* Fast Image Processing with Fully-Convolutional Networks

,

ICCV 2017

Code

Some popular software repositories that came out of our research. * CARLA: An Open Urban Driving Simulator * Open3D: A Modern Library for 3D Data Processing * OpenBot: Turning Smartphones into Robots * Learning to See in the Dark * Temporal Convolutional Networks * Tracking Objects as Points * Direct Sparse Odometry * Photographic Image Synthesis * Dilated Residual Networks

Talks

* Further Towards Photorealism (May 2021) * Towards Photorealism (September 2020) * Autonomous Driving: The Way Forward (July 2020) * Beyond convolutional networks (June 2020) * Towards machines that see in the real world (June 2020) * Learning to drive (April 2018) * Doing (Good) Research (June 2018)

Interviews

* Gradient Dissent (Lukas Biewald, January 2021) * Humans of AI: Stories, Not Stats (Devi Parikh, September 2020)

Images

Stable View Synthesis ".)" class="thickbox"

rel="set_3">

OpenBot: Turning Smartphones into Robots ".)" class="thickbox"

rel="set_3">

Learning Quadrupedal Locomotion over Challenging Terrain ".)" class="thickbox"

rel="set_3">

Multiscale Deep Equilibrium Models ".)" class="thickbox"

rel="set_3">

Deep Drone Acrobatics ".)" class="thickbox"

rel="set_3">

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer ".)" class="thickbox"

rel="set_3">

Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing

".)"

class="thickbox" rel="set_3"> Learning Agile and Dynamic Motor Skills for Legged Robots ".)" class="thickbox"

rel="set_3">

Deep Drone Racing: Learning Agile Flight in Dynamic Environments

".)"

class="thickbox" rel="set_3"> Semi-parametric Image Synthesis ".)" class="thickbox"

rel="set_3">

Learning to See in the Dark

".)"

class="thickbox" rel="set_3"> End-to-end Driving via Conditional Imitation Learning

".)"

class="thickbox" rel="set_3"> Photographic Image Synthesis with Cascaded Refinement Networks

".)"

class="thickbox" rel="set_3"> Playing for Benchmarks

".)"

class="thickbox" rel="set_3"> Fast Image Processing with Fully-Convolutional Networks

".)"

class="thickbox" rel="set_3"> Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction

".)"

class="thickbox" rel="set_3"> Playing for Data: Ground Truth from Computer Games

".)"

class="thickbox" rel="set_3"> Feature Space Optimization for Semantic Video Segmentation ".)" class="thickbox"

rel="set_3">

Single-View Reconstruction via Joint Analysis of Image and Shape

Collections ".)"

class="thickbox" rel="set_3"> Robust Reconstruction of Indoor Scenes ".)" class="thickbox"

rel="set_3">

Learning to Propose Objects ".)" class="thickbox"

rel="set_3">

Color Map Optimization for 3D Reconstruction with Consumer Depth

Cameras ".)"

class="thickbox" rel="set_3"> Dense Scene Reconstruction with Points of Interest

".)"

class="thickbox" rel="set_3"> A Probabilistic Model for Component-Based Shape Synthesis

".)"

class="thickbox" rel="set_3"> An Algebraic Model for Parameterized Shape Editing ".)" class="thickbox"

rel="set_3">

Efficient Inference in Fully Connected CRFs with Gaussian Edge

Potentials ".)"

class="thickbox" rel="set_3"> Computer-Generated Residential Building Layouts

".)"

class="thickbox" rel="set_3">

Gesture Controllers

".)"

class="thickbox" rel="set_3"> Probabilistic Reasoning for Assembly-Based 3D Modeling ".)" class="thickbox"

rel="set_3">

Latest News

TWO PAPERS ACCEPTED TO CVPR 2021 Two papers were accepted to the Conference on Computer Vision and Pattern Recognition (CVPR) : Self-supervised Geometric Perception and Stable View Synthesis

.

Self-supervised Geometric Perception was selected for oral presentation at the conference.

Other News

PAPER ACCEPTED TO ICRA 2021 Paper accepted to ICRA 2021 PAPER ACCEPTED TO ICLR 2021 Paper accepted to ICLR 2021 PAPER PUBLISHED IN SCIENCE ROBOTICS AND FEATURED ON THE COVER Paper published in Science Robotics and featured on the cover PAPER ACCEPTED TO NEURIPS 2020 Paper accepted to NeurIPS 2020 THREE PAPERS ACCEPTED TO ECCV 2020 Three papers accepted to ECCV 2020

* Home

* /

* News

* /

* Projects

* /

* Publications

* /

* Lab

* /

* Teaching

* /

* Contact

* /

COPYRIGHT 2021. ALL RIGHTS RESERVED.

loading