Hi, I am Steffen Schotthöfer!

I am the Householder Fellow in Mathematics at the Oak Ridge National Laboratory, USA. My current research is centered around AI for Science, with focus on low-rank approximation methods for high dimensional PDEs and neural networks. My goal is to create increasingly robust and efficient methods for complex simulations, in particular for plasma and nuclear physics.

News

Paper out: April. 26 2024 Github

The main challenge of large-scale numerical simulation of radiation transport is the high memory and computation time requirements of discretization methods for kinetic equations. ... In this work, we derive and investigate a neural network-based approximation to the entropy closure method to accurately compute the solution of the multi-dimensional moment system with a low memory footprint and competitive computational time. We extend methods developed for the standard entropy-based closure to the context of regularized entropy-based closures. The main idea is to interpret structure-preserving neural network approximations of the regularized entropy closure as a two-stage approximation to the original entropy closure. We conduct a numerical analysis of this approximation and investigate optimal parameter choices. Our numerical experiments demonstrate that the method has a much lower memory footprint than traditional methods with competitive computation times and simulation accuracy. We present numerical comparisons to existing conservative integrators and discuss advantages and disadvantages in the full paper.

Read more

Paper out: Feb. 26 2024

Can we use operator learning to accellerate PDE simulations without sacrificing structural properties of the underlying operators? ... Yes! We investigate the capabilities of the Deep Operator network (DeepONet) approach to modelling the high dimensional collision operator of the linear kinetic equation. This integral operator has crucial analytical structures that a surrogate model, e.g., a DeepONet, needs to preserve to enable meaningful physical simulation. We propose several DeepONet modifications to encapsulate essential structural properties of this integral operator in a DeepONet model. To be precise, we adapt the architecture of the trunk-net so the DeepONet has the same collision invariants as the theoretical kinetic collision operator, thus preserving conserved quantities, e.g., mass, of the modeled many-particle system. Further, we propose an entropy-inspired data-sampling method tailored to train the modified DeepONet surrogates without requiring an excessive expensive simulation-based data generation in the full paper.

Read more

Paper out: Nov. 17 2023 Github

One key question when using DLRA methods is the construction of robust time integrators that preserve the invariances and associated conservation laws of the original problem. ... Numerical simulations of kinetic problems can become prohibitively expensive due to their large memory footprint and computational costs. A method that has proven to successfully reduce these costs is the dynamical low-rank approximation (DLRA). In this work, we demonstrate that the augmented basis update & Galerkin integrator (BUG) preserves solution invariances and the associated conservation laws when using a conservative truncation step and an appropriate time and space discretization. We present numerical comparisons to existing conservative integrators and discuss advantages and disadvantages in the full paper.

Read more

Paper published: Sept. 15 2023 Github

Gas dynamic simulations that span multiple flow regimes are a challenging problem in high-altitude aerospace, turbo-machinery and propulsion engines. ... Shock regions require expensive, high resolution kinetic schemes, but are local phenomena. We build a physics informed neural network based detector of shock regions using only coarse grained information. Based on this flow-regime classification, solvers with different physical resolutions are employed to enable a robust, but efficient hybrid simulation. The work is published in the Journal of Computational Physics and the preprint is available on Arxiv.

Read more

New Position: Sep. 3rd 2023

I'm excited to start my new position as Householder Fellow at the Oak Ridge National Laboratory. ... In the next years my research is focussed on improving dynamical low-rank methods for neural network training and investigating its applications.

Read more

Paper out: May 30 2023

The computing cost and memory demand of deep learning pipelines have grown fast in recent years and thus a variety of pruning techniques have been developed to reduce model parameters. ... The majority of these techniques focus on reducing inference costs by pruning the network after a pass of full training. A smaller number of methods address the reduction of training costs, mostly based on compressing the network via low-rank layer factorizations. Despite their efficiency for linear layers, these methods fail to effectively handle convolutional filters. In this work, we propose a low-parametric training method that factorizes the convolutions into tensor Tucker format and adaptively prunes the Tucker ranks of the convolutional kernel during training. Leveraging fundamental results from geometric integration theory of differential equations on tensor manifolds, we obtain a robust training algorithm that provably approximates the full baseline performance and guarantees loss descent. A variety of experiments against the full model and alternative low-rank baselines are implemented, showing that the proposed method drastically reduces the training costs, while achieving high performance, comparable to or better than the full baseline, and consistently outperforms competing low-rank approaches. Read the full paper on arxiv.

Read more

Publications

2024

• Structure-preserving neural networks for the regularized entropy-based closure of the Boltzmann moment system
  Steffen Schotthöfer, M. Paul Laiu, Martin Frank, Cory Hauck
  ArXiv preprint

• Structure-Preserving Operator Learning: Modeling the Collision Operator of Kinetic Equations
  Jae Yong Lee, Steffen Schotthöfer, Tianbai Xiao, Sebastian Krumscheid, Martin Frank
  ArXiv preprint

2023

• Conservation properties of the augmented basis update & Galerkin integrator for kinetic problems
  Lukas Einkemmer, Jonas Kusch, Steffen Schotthöfer
  ArXiv preprint Github
• Rank-adaptive spectral pruning of convolutional layers during training
  Emanuele Zangrando, Steffen Schotthöfer, Gianluca Ceruti, Jonas Kusch, Francesco Tudisco
  ArXiv preprint
• Synergies between Numerical Methods for Kinetic Equations and Neural Networks
  Steffen Schotthöfer
  Dissertation

2022

• Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations
  Steffen Schotthöfer, Emanuele Zangrando, Jonas Kusch, Gianluca Ceruti, Francesco Tudisco
  ArXiv preprint 2205.13571, accepted for NeurIPS2022 Github
• Structure Preserving Neural Networks: A Case Study in the Entropy Closure of the Boltzmann Equation
  Steffen Schotthöfer, Tianbai Xiao, Martin Frank, Cory Hauck
  ICML 2022 Github
• KiT-RT: An extendable framework for radiative transfer and therapy
  Jonas Kusch, Steffen Schotthöfer, Pia Stammer, Jannick Wolters, Tianbai Xiao
  ArXiv preprint 2205.08417 Github
• Predicting continuum breakdown with deep neural networks
  Tianbai Xiao, Steffen Schotthöfer, Martin Frank
  ArXiv preprint 2203.02933 Github

2021

• A structure-preserving surrogate model for the closure of the moment system of the Boltzmann equation using convex deep neural networks
  Steffen Schotthöfer, Tianbai Xiao, Martin Frank, Cory Hauck
  AIAA AVIATION 2021 FORUM Github

2020

• Regularization for Adjoint-Based Unsteady Aerodynamic Optimization Using Windowing Techniques
  Steffen Schotthöfer, Beckett Y. Zhou, Tim Albring, Nicolas R. Gauger
  AIAA Journal Github

2018

• A Numerical Comparison of Consensus-Based Global Optimization to other Particle-based Global Optimization Schemes
  Claudia Totzeck, René Pinnau, Sebastian Blauth, Steffen Schotthöfer
  Proceedings in Applied Mathematics and Mechanics

Community and Volunteering

• Reviewer for SIAM Multiscale Methods and Simulation
• Reviewer for the NeurIPS 2024 Conference
  
• Reviewer for the ICML 2024 Conference
• Reviewer for the NeurIPS 2023 Conference
• Organization of the International Workshop on Moment Methods in Kinetic Theory IV

Conferences

• Neural Network Training with Dynamical Low-Rank Inspired Optimizers
  SciCADE 2024
  July 2024
• Conservation Properties of the Augmented Basis Update and Galerkin Integrator for Kinetic Problems
  SIAM Annual Meeting 2024
  July 2024
• Memory and time efficient neural network training via dynamical low-rank approximation
  GAMM Annual Meeting 2024
  March 2024
• Regularized, structure-preserving neural networks for the minimal entropy closure of the Boltzmann moment system
  GAMM Annual Meeting 2024
  March 2024
• Structure Preserving Neural Networks for Moment Method Acceleration
  International Workshop on Moment Methods in Kinetic Theory IV
  April 2023
• Regularized, Structure-preserving Neural Networks for the Minimal Entropy Closure of the Boltzmann Moment System
  SIAM CSE 2023 Conference
  February 2023
• Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations
  NeurIPS 2022 Conference
  November 2022
• Regularized, Structure-preserving Neural Networks for the Minimal Entropy Closure of the Boltzmann Moment System
  Theoretical Foundations of Deep Learning - Annual Meeting, DFG Priority Programme
  November 2022
• Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations
  AI Hub @ Karlsruhe | 2022
  October 2022
• Neural network-based, structure-preserving entropy closures for the Boltzmann moment system
  GIMC SIMAI YOUNG
  September 2022
• Structure Preserving Neural Networks: A Case Study in the Entropy Closure of the Boltzmann Equation
  ICML Conference
  July 2022
• A structure-preserving surrogate model for the closure of the moment system of the Boltzmann equation using convex deep neural networks
  AIAA AVIATION 2021 FORUM
  August 2021
• Windowing Regularization Techniques for Unsteady Aerodynamic Shape Optimization
  AIAA AVIATION 2020 FORUM
  June 2020
• Windowing Regularization Techniques for Unsteady Aerodynamic Shape Optimization
  SU2 Conference 2020 , (Recorded talk)
  June 2020

Selected Talks

• Dynamical Low-Rank Approximation for Kinetic Equations and Neural Networks
  Lawrence Berkeley National Laboratories, CA, USA
  December 2023
• Dynamical Low-Rank Approximation for Kinetic Equations and Neural Networks
  Lawrence Berkeley National Laboratories, CA, USA
  December 2023
• Dynamical Low-Rank Compression of Neural Networks
  ICL Seminar, University of Tennessee, Knoxville, USA
  September 2023
• Low-rank lottery tickets: Finding efficient low-rank neural networks via matrix differential equations
  MODUS Seminar, University Bayreuth, Germany
  January 2023
• Low-rank lottery tickets: Finding efficient low-rank neural networks via matrix differential equations
  Scientific Computing Group, RPTU Kaiserslautern, Germany
  December 2022
• Low-rank lottery tickets: Finding efficient low-rank neural networks via matrix differential equations
  Numerical Analysis and Scientific Computing Group, University of Innsbruck, Austria
  August 2022
• Structure Preserving Neural Network Based Entropy Closures for the Boltzmann Moment System
  Oak Ridge National Lab, TN, USA
  April 2022
• Structure Preserving Neural Network Based Entropy Closures for the Boltzmann Moment System
  KIT, Karlsruhe, Germany
  February 2022
• Hybrid machine learning and numerical methods for radiative transport equations
  KIT, Karlsruhe, Germany
  December 2020

Software projects

KiT-RT Github

The main focus of the KiT-RT software suite is on radiotherapy planning for cancer treatment and investigation of various research questions in the field of radiative transfer. This goal is supported by an easily extendable code structure that allows for straightforward implementation of additional methods and techniques. The KiT-RT framework is a high-performance open source C++ based platform for radiation transport, available on Github with documentation on ReadTheDocs. The software-paper can be found on Arxiv

SU2 Github

Drag reduction of airplane wings is crucial for fuel efficient flight. We use windowing regularization to build a robust PDE constrained optimization for unsteady flows. On the NACA0012 Airfoil profile with an unsteady, turbulent flow at high angle of attack, we've achieved 30% drag reduction compared to the unregularized baseline optimization. The method is embedded in the open-source, high performance multi-physics software SU2. Try it yourself with the SU2 tutorial.

This work was awarded 1st place at the Multidisciplinary Design Optimization Student Paper Competititon at the AIAA aviation forum 2020.