Tutorial Sessions

Alex Olshevsky, Associate Professor in Electrical and Computer Engineering, Boston University, Boston, MA

Title: Analyzing Reinforcement Learning as Gradient Descent

Abstract: This tutorial will present some recent work drawing connections between the optimization literature on gradient descent and reinforcement learning (RL). By the end of the tutorial, you will be able to explain how proofs in the optimization literature can sometimes be carried over verbatim to the study of algorithms RL. Some open problems will be discussed where it is possible that the optimization perspective can play a substantial role.  The tutorial will be self-contained and will assume no knowledge or background in RL or convex optimization; only a basic understanding of convexity is required. Topics covered include:

1. A gentle introduction to reinforcement learning. 

2. Interlude on gradient descent. Analysis of temporal difference methods using ideas from gradient descent. 

3. Policy gradient and actor-critic. Actor-critic as bilevel optimization. 

4. Neural networks. Bounded nonlinearity, connections to inexact gradient descent. 

5. Open questions. 

Bio:   Alex Olshevsky received a B.S. degree in applied mathematics and a B.S. degree in electrical engineering from the Georgia Institute of Technology, Atlanta, GA, USA, both in 2004 and the M.S. and Ph.D. degrees in electrical engineering and computer science from the Massachusetts Institute of Technology, Cambridge, MA, USA, in 2006 and 2010, respectively. He was a postdoctoral scholar at Princeton University from 2010 to 2012, and an Assistant Professor at the University of Illinois at Urbana-Champaign from 2012 to 2016. He is currently an Associate Professor with the ECE department at Boston University. Dr. Olshevsky is a recipient of the NSF CAREER Award, the Air Force Young Investigator Award, the INFORMS Computing Society Prize for the best paper on the interface of operations research and computer science, a SIAM Award for the annual paper from the SIAM Journal on Control and Optimization chosen to be reprinted in SIAM Review, and an IMIA award for best paper on clinical medical informatics in 2019.

Victoria Kostina, professor of electrical engineering and of computing and mathematical sciences at Caltech, Pasadena, CA

Title:  Information bottlenecks in control systems

Abstract:  Classical information theory created by Claude Shannon in 1948 describes optimal tradeoffs in data compression and transmission that are achievable in the limit of long transmission times or data sequences. In classical information theory, the message is divorced from its context, and the sole task of the code designer is to reproduce, quoting Shannon, “at one point either exactly or approximately a message selected at another point”. While classical information theory enabled the present era of global connectivity, it does not supply ready answers for remotely controlled dynamical systems. This is because they are extremely delay sensitive, so that coding over long blocks of observed data might not be feasible. Furthermore, information exchanges in such human-to-machine or machine-to-machine communication systems are performed with the purpose of performing a task, e.g., to precisely control the arm of a robotic surgeon, thus, the context, or the goal of communication, plays a role in code design. In other words, the job of the code designer is being expanded from just protecting the message from channel noise to also deciding what information about the observed data to send, given what the receiver already knows, in order to achieve the goal. To be successful, a code designer must not only understand the limits of feedback communication but also those of the control system of interest.  In this tutorial, I will show how to cast the above challenges in mathematical language and introduce the information-theoretic tools that have proven useful to tackle them;  using toy examples, I will highlight recent research progress in control-information tradeoffs, and I will conclude with concrete open problems.   

Bio: Victoria Kostina is a professor of electrical engineering and of computing and mathematical sciences at Caltech. She received a bachelor’s degree from Moscow Institute of Physics and Technology (MIPT) in 2004, the master’s degree from the University of Ottawa in 2006, and the Ph.D. degree from Princeton University in 2013.  During her studies at MIPT, she was affiliated with the Institute for Information Transmission Problems of the Russian Academy of Sciences. Her research interests lie in information theory, coding, communications, learning, and control.  She has served as an Associate Editor for IEEE Transactions of Information Theory, and as a Guest Editor for the IEEE Journal on Selected Areas in Information Theory.  She received the Natural Sciences and Engineering Research Council of Canada postgraduate scholarship during 2009–2012, Princeton Electrical Engineering Best Dissertation Award in 2013, Simons-Berkeley research fellowship in 2015 and the NSF CAREER award in 2017.