Mathematical Sciences Department
Virtual Numerical Methods Seminar
Speaker: Shuhao Cao, University of Missouri, Kansas City
Monday, December 11, 2023
11:00 am – 12:00 pm
Zoom Meeting ID: 929 6655 1992
Title: Structure-conforming Operator Learning via Transformers
Abstract: GPT, Stable Diffusion, AlphaFold 2, etc., all these state-of-the-art deep learning models use a neural architecture called "Transformer". Since the emergence of "Attention Is All You Need", Transformer is now the ubiquitous architecture in deep learning. At Transformer's heart and soul is the "attention mechanism". In this talk, we shall give a specific example of a fundamental but critical question: whether and how one can benefit from the theoretical structure of a mathematical problem to develop task-oriented and structure-conforming deep neural networks? An attention-based deep direct sampling method is proposed for solving Electrical Impedance Tomography (EIT), a class of boundary value inverse problems. Progresses within different communities will be briefed to answer some open problems on the mathematical properties of the attention mechanism in Transformers. This is joint work with Ruchi Guo (UC Irvine) and Long Chen (UC Irvine).