Francesco Pittaluga

Electrical Engineering PhD Student
University of Florida
f dot pittaluga at ufl dot edu


News

  • 01/07/19 - Presenting Learning Privacy Preserving Encodings through Adversarial Training at WACV 2019
  • 11/01/18 - Speaking at Magic Leap's Advanced Technologies Lab in Seattle, WA
  • 10/29/18 - Participating in Microsoft Research PhD Summit
  • 06/09/18 - Awarded Microsoft Research Dissertation Grant for work on Privacy Preserving Computational Cameras
  • 06/14/18 - Joining Microsoft Research as a Research Intern
  • 11/01/17 - Pre–Capture Privacy Cameras for Small Vision Sensors published in PAMI
  • 07/05/17 - Presenting on Privacy Preserving Computational Cameras at CVPR Workshop: Challenges and Opportunities for Privacy and Security
  • 05/20/17 - Joining Magic Leap's Advanced Technologies Lab as an Intern
  • 09/02/16 - Joining Toyota Technological Institute of Technology as a Graduate Researcher
  • 07/15/16 - Presenting on Pre–Capture Privacy Cameras at Safe Autonomous Cyber Physical Systems Workshop 2016
  • 07/05/16 - Speaking on Pre–Capture Privacy Cameras at 9th Annual DNDO ARI Conference in Atlanta, GA
  • 05/13/16 - Speaking on Sensor-level Privacy for Thermal Cameras at ICCP 2016
  • 06/08/15 - Presenting Privacy Preserving Optics for Miniature Vision Sensors at CVPR 2015

Bio

I am a fifth year doctoral student in Electrical Engineering at the University of Florida, where I work at the FOCUS Lab under the direction of Sanjeev J. Koppal, and the recipient of a 2018-2019 Microsoft Research Dissertation Grant. My research interests include computer vision, machine learning and computational photography. Over the last three years, I've interned at the Toyota Technological Institute of Technology at Chicago, where I worked with Ayan Chakrabarti; Magic Leap's Advanced Technologies Lab in Seattle, where I worked with Laura Trutoiu and Brian Schowengerdt; and Microsoft Research, where I worked with Sudipta Sinha and Sing Bing Kang. Prior to beginning my doctoral studies, I attended Tufts University, where I received a B.S. in Electrical Engineering with a second major in Computer Science and worked as an undergraduate researcher under the direction of Karen Panetta. During this time, I also interned at GE Intelligent Platforms and participated in the NSF Research Experience for Undergraduates (REU) Program at Florida International University, in my hometown of Miami, FL. My mentor for the REU program was Niki Pissinou.

Tufts Logo
GE Logo
UF Logo
TTIC Logo
Magic Leap Logo
Microsoft Logo

Tufts University
2010-2014

GE Intelligent Platforms
Summer 2013

University of Florida
2014-Present

TTIC at Chicago
Fall 2016

Magic Leap
Advanced Tech. Lab
Summer 2017

Microsoft Research
Summer 2018


Publications

Under Review, 2019

Revealing Scenes by Inverting Structure from Motion Reconstructions
Francesco Pittaluga, Sanjeev J. Koppal, Sing Bing Kang, Sudipta Sinha

Under Review, 2019

Privacy Preserving Action Recognition using Coded Aperture Videos
Zihao W. Wang, Vibhav Vineet, Francesco Pittaluga, Sudipta Sinha, Sing Bing Kang

WACV 2019

Learning Privacy Preserving Encodings through Adversarial Training
Francesco Pittaluga, Sanjeev J. Koppal, Ayan Chakrabarti

We present a framework to learn privacy preserving encodings of images that inhibit inference of chosen private attributes, while allowing recovery of other desirable information. Rather than simply inhibiting a given fixed pretrained estimator, our goal is that an estimator be unable to learn to accurately predict the private attributes even with knowledge of the encoding function. We use a natural adversarial optimization-based formulation for this training the encoding function against a classifier for the private attribute, with both modeled as deep neural networks. The key contribution of our work is a stable and convergent optimization approach that is successful at learning an encoder with our desired properties maintaining utility while inhibiting inference of private attributes, not just within the adversarial optimization, but also by classifiers that are trained after the encoder is fixed. We adopt a rigorous experimental protocol for verification wherein classifiers are trained exhaustively till saturation on the fixed encoders. We evaluate our approach on tasks of real-world complexity learning high-dimensional encodings that inhibit detection of different scene categories and find that it yields encoders that are resilient at maintaining privacy.

Paper »

PAMI 2017, CVPR 2015

Pre-Capture Privacy for Small Vision Sensors
Francesco Pittaluga and Sanjeev J. Koppal

The next wave of micro and nano devices will create a world with trillions of small networked cameras. This will lead to increased concerns about privacy and security. Most privacy preserving algorithms for computer vision are applied after image/video data has been captured. We propose to use privacy preserving optics that filter or block sensitive information directly from the incident light-field before sensor measurements are made, adding a new layer of privacy. In addition to balancing the privacy and utility of the captured data, we address trade-offs unique to miniature vision sensors, such as achieving high-quality field-of-view and resolution within the constraints of mass and volume. Our privacy preserving optics enable applications such as depth sensing, full-body motion tracking, people counting, blob detection and privacy preserving face recognition. While we demonstrate applications on macro-scale devices (smartphones, webcams, etc.) our theory has impact for smaller devices.

ICCP 2016

Sensor-level Privacy for Thermal Cameras
Francesco Pittaluga, Aleksandar Zivkovic and Sanjeev J. Koppal

As cameras turn ubiquitous, balancing privacy and utility becomes crucial. To achieve both, we enforce privacy at the sensor level, as incident photons are converted into an electrical signal and then digitized into image measurements. We present sensor protocols and accompanying algorithms that degrade facial information for thermal sensors, where there is usually a clear distinction between humans and the scene. By manipulating the sensor processes of gain, digitization, exposure time, and bias voltage, we are able to provide privacy during the actual image formation process and the original face data is never directly captured or stored. We show privacy-preserving thermal imaging applications such as temperature segmentation, night vision, gesture recognition and HDR imaging.

Paper »

TePRA 2013

Facial Recognition Using Human Visual System Algorithms for Robotic and UAV Platforms
Nicholas Davis, Francesco Pittaluga and Karen Panetta

First responders' ability to respond rapidly to emergency situations is limited by a lack of real time intelligence. To ensure the safety of the responders, the situation must first be evaluated for dangerous conditions including life-threatening hazards. Live visual feeds let remote experts gauge the safety levels and assess damages of a situation, but do not perform adequately when images are captured in poor lighting or in harsh environmental conditions. We present a novel approach that leverages HVS-based (Human Visual System) object detection in combination with low-cost commercial off-the-shelf UAVs, to deliver efficient real time image enhancement and detection. This approach enables our system to deliver timely information in low visibility environments making it ideal for aiding first responders in their search for critical objects such as wounded victims, human bodies and threat objects.

Paper »


Patents

US Patent App. 15/577,019

Pre-capture De-identification (PCDI) Imaging System and PCDI Optics Assembly
Sanjeev Koppal and Francesco Pittaluga

A de-identification assembly comprising an object tracking sensor to track features of an object; and a mask generator to produce rays of light in response to the tracked features of the object, the rays of light representing a de-identification mask of the object. The assembly includes a beamsplitter having a first side configured to receive rays of light representing the object and a second side configured to receive the rays of light of the mask from the mask generator. The beamsplitter produces a composite image of the object superimposed with the de-identification mask to anonymize an image of the object. A system including the de-identification assembly and a method are also provided.

Patent »

US Patent App. 15/561,251

Optical Privatizing Device, System and Method of Use
Sanjeev Koppal and Francesco Pittaluga

Embodiments are directed to an optical privatizing device, system and methods of use. The device includes a removable frame removably attachable to a sensor housing. A device includes a blurring lens coupled to the removable frame and configured to optically modify light passing to a depth sensor wherein the optical modified light has a privatizing blur level to neutralize a profile of an object sensed by the depth sensor within a working volume of the depth sensor to an unidentifiable state while maintaining a depth parameter sensed by the depth sensor.

Patent »


Side Projects

Frogger Web App

Rules: The game starts with five frogs, which are counted as the player's lives. Losing all five frogs results in the end of the game. The objective of the game is to guide each frog to one of the designated spaces at the top of the screen. The frog starts at the bottom of the screen. The player must guide the frog between opposing lanes of traffic to avoid becoming roadkill, which results in a loss of one life. The upper portion of the screen consists of a river with logs and turtles, all moving horizontally across the screen. By jumping on swiftly moving logs and the backs of turtles the player can guide their frog to safety. The player may catch bugs which appear periodically for bonuses.

Ten-GPU Server Build

I put together this 10-GPU server box (with the help of Jason Kawaja) for use in machine learning research at the FOCUS Lab at the University of Florida. After extensive use, it is still running wonderfully. I highly recommend this setup to anyone interested in building their own GPU server.

Specifications:

GPUs: One NVIDIA TitanX, Nine NVIDIA 1080ti
CPUs: Two Intel Xeon E5-2680 v4
RAM: 512GB (16x32GB) DDR4 2400
Storage: 10TB (5x2TB) SSD
Cables: 8 Pin Male to Dual 2x8 Pin Male PCI Express
Motherboard, Case, Fans, Power Supplies: Supermicro SYS-4028GR-TRT2