BiMed-Lab Group Summer Conference 2024200 years old and still relevant: The Gauss-Seidel principle.Steffen Plunder19th September 2024Kyoto
Applied math is about connecting ideas:ModellingBiologyMedicineAnalysisNumericsOptimisationComputer SciencePhysicsChemistryStatisticsData science
An old challenge: Solving linear equationsLinear systemsCarl Friedrich Gauss (and earlier mathematicians!)A modern challenge: Solve nonlinear partial differential equations on GPUs nonlinear, high-dimensional, lots of iterations and we need it fast...Solve
OutlineLinear: The classical Gauss-Seidel principleOptimisation: Nonlinear Gauss-SeidelPDEs on GPUs: Graph coloring!Possible applications in modelling?
SourcesMainly inspired by a recent article in compute science by Anka He Chen (July 2024):Old textbooks by Werne C. Rheinboldt, James M. Ortega, ...:Related to my KAKENHIE project aroundthe math of NVIDIA Warp
Example of the "Gauss-Seidel Principle" in ActionChen, A. H., Liu, Z., Yang, Y. & Yuksel, C. Vertex Block Descent. ACM Transactions on Graphics (2024).Chen, Y. et al. Position-Based Nonlinear Gauss-Seidel for Quasistatic Hyperelasticity. ACM Transactions on Graphics (2024).
History of the Gauss-Seidel method in computer science (aka PBD)(2006) Müller, Heidelberger, Hennix, Ratcliff Position Based Dynamics (2013) Macklin, Müller: Position Based Fluids (2015) Wang et al: Chebyshev Semi-Iterative Approach for Accelerating Projective and Position-based Dynamics(2016) Macklin, Müller: XPBD(2019) Macklin et al: Small Steps in Physics Simulation(2024) Anka Chen et al: Vertex Block Decent
Relation to my researchThm. [SP, Sara Merino (2023) arxiv]For first order dynamics, Position-Based Dynamics converges (uniformly).computational costShows that non-smooth , projected Gauss-Seidelconverges for many biological models...
Applied mathematician (PhD 2024) Anka He ChenGauss-Seidel is still relevant...few weeks agoMiles Marklin(NVIDIA Warp)Jen-Hsun Huang(NVIDIA CEO)
Part I:The origin story
Letter from 1823 from Gauss to Gerling How to determine long distances withoutGoogle Maps (in 1823)?!(my home)
Linear Algebra before computers ** Historically "computer" was an occupation: "one who computes". See also the movie Hidden Figures (2016)?~12km ± 1km8km ± 1km4km ± 0.5km (Normal Equation)Letter to Gerling showing a quicker way to compute angles and distances using Gauss' new idea!
Direct Methods: Gauss Elimination"Issue"Gauss elemination steps operate on the full matrix A.What if the matrix is large (or the computer is lazy)? It does not preserve sparsity!
Gauss' trick (in the letter from 1823)Solve equations one-by-one:Initial guess repeatFixed-point iteration!
Part II:Nonlinear Optimisation
Nonlinear Optimisation: Isn't it always Gradient Descent? But is a very large vector in PDE settings...Can we apply the Gauss-Seidel principle here?!
Gauss-Seidel "Style" Nonlinear Optimisation We never need to construct a large vector/need to solve a large!Only sequential solution of univariable nonlinear equations.But the algorithm is very sequential, not parallel!
Red-Black Gauss-Seidel 1234 Solve black (parallel)!Copy values.Solve red (parallel)!Note for later:We could do the same with more colors...
Part III:Gauss-Seidel for solving PDEs on GPUs
Typical solution strategy (implicit methods)PDE Space-discretisation Assemble large, sparse matrix Solve with iterative methods
CPUs and GPUs (very simplified)CPUThreadsThreadsThreadsShared Memory (RAM)Shared CacheGPUGroupThreadsThreadsThreadsShared Memory (GRAM)Shared CacheGroupThreadsThreadsThreadsShared CacheGroupThreadsThreadsThreadsShared CacheIssue:Assembly requires a lot of sharedmemory usage! Slow on GPU!
Typical solution strategy for nonlinear PDEs (implicit methods)PDE Solve with Newton method: LinearizeAssemble large, sparse matrix Solve linear system
The Gauss-Seidel principle!Why do we solve for all variables at once? Instead, we could just solve many smaller problemsfor each degree of freedom!
Variational form of PDEs on meshesvertex (degree of freedom) Total potential energyLocal deformation energy of vertex Gauss-Seidel Principle:
Vertex Block Decent (2024)Represent shape as ameshCompute Graph Coloringfor all vertices.Formulate PDE in variational form("physical systems minimize their energy/action") Apply parallel Gauss-Seidel using graph colors!
Advantage over typical Newton-style strategyPDEdiscretize globallylarge sparse system(each iteration is expensive)discretize locally(per vertex)solve univariate problems(cheap, but less accurate)CPU ᡣ𐭩GPU ᡣ𐭩(Some other tricks to speed up convergence and improve stability...)
Part IV:Applications and Next Steps
Github package: SpatialHashTables.jl (GPU accelerated particle simulations)Missing... GPU graph coloring.Claudia, Sara MerinoMichèle RomanosTsutsumi-san,AntoineMy current models... only spherical/elliptic cells!Why notsimulate membranes?
Vertex Block Decent is basically half of my Kakenhi!
Convergence?Numerical testsMathematical theoryConvergence theorems not adapted to VBD yet,but very close...Contact forcesSpecifically for biology applications:One could combine PBD with VBD!
ThanksThe Gauss-Seidel Take-Away1. Why not solve many simple problems, instead of one large one?2. Efficienacy = Accuracy / Runtime