Awesome resources on normalizing flows.
This page lists names, links and short descriptions. The original list on GitHub is the source and belongs to its authors.
by Klein, Noé Boltzmann Generators, a machine learning method, generate equilibrium samples of molecular systems by learning a transformation from a simple prior distribution to the target Boltzmann distribution via normalizing flows. Recently, flow matching has been used to train Boltzmann…
by Kallapa, Nagar et al. propose a k×k convolutional layer and Deep Normalizing Flow architecture which i) has a fast parallel inversion algorithm with running time O(nk^2) (n is height and width of the input image and k is kernel size), ii) masks the minimal amount of learnable parameters in a…
by Ahn, Kim et al. This work proposes the monotone formulation to overcome the issue of the Lipschitz constants in previous ResNet-based normalizing flows using monotone operators and provides an in-depth theoretical analysis. Furthermore, this work constructs an activation function called…
by Postels, Danelljan et al. The invertibility constraint of NFs imposes limitations on data distributions that reside on lower dimensional manifolds embedded in higher dimensional space. This is often bypassed by adding noise to the data which impacts generated sample quality. This work generates…
by Wehenkel, Louppe This work revisits coupling and autoregressive transformations as probabilistic graphical models showing they reduce to Bayesian networks with a pre-defined topology. From this new perspective, the authors propose the graphical normalizing flow, a new invertible transformation…
by Feng, Xu et al. Proposes a novel non-autoregressive deep learning model, called Multi-scale Attention Normalizing Flow(MANF), where one integrates multi-scale attention and relative position information and the multivariate data distribution is represented by the conditioned normalizing flow.
by Gabrié, Rotskoff et al. Markov Chain Monte Carlo (MCMC) algorithms struggle with sampling from high-dimensional, multimodal distributions, requiring extensive computational effort or specialized importance sampling strategies. To address this, an adaptive MCMC approach is proposed, combining…
by Satorras, Hoogeboom et al. Introduces equivariant graph neural networks into the normalizing flow framework which combine to give invertible equivariant functions. Demonstrates their flow beats prior equivariant models and allows sampling of molecular configurations with positions, atom types…
by Gabrié, Rotskoff et al. Normalizing flows have potential in Bayesian statistics as a complementary or alternative method to MCMC for sampling posteriors. However, their training via reverse KL divergence may be inadequate for complex posteriors. This research proposes a new training approach…
by Nagar, Dufraisse et al. Seeks to improve expensive convolutions. They investigate the conditions for when 3x3 convolutions are invertible under which conditions (e.g. padding) and saw successful speedups. Furthermore, they developed a more expressive, invertible Quad coupling layer. [Code]
by Trockman, Kolter Parametrizes the multichannel convolution to be orthogonal via the Cayley transform (skew-symmetric convolutions in the Fourier domain). This enables the inverse to be computed efficiently. [Code]
by Goliński, Lezcano-Casado et al. Parametrizes the 1x1 convolution via the exponential map and the Cayley map. They demonstrate an improved optimization for the Sylvester normalizing flows.
by Rasul, Sheikh et al. Models the multi-variate temporal dynamics of time series via an autoregressive deep learning model, where the data distribution is represented by a conditioned normalizing flow. [OpenReview.net] [Code]
by Bhattacharyya, Straehle et al. Introduce a Haar wavelet-based block autoregressive model.
by Dolatabadi, Erfani et al. An adversarial attack method on image classifiers that use normalizing flows. [Code]
by Nielsen, Jaini et al. They present a generalized framework that encompasses both Flows (deterministic maps) and VAEs (stochastic maps). By seeing deterministic maps x = f(z) as limiting cases of stochastic maps x ~ p(x|z), the ELBO is reinterpreted as a change of variables formula for the…
by Kirichenko, Izmailov et al. This study how traditional normalizing flow models can suffer from out-of-distribution data. They offer a solution to combat this issue by modifying the coupling layers. [Tweet] [Code]
by Köhler, Klein et al. Shows that distributions generated by equivariant NFs faithfully reproduce symmetries in the underlying density. Proposes building blocks for flows which preserve typical symmetries in physical/chemical many-body systems. Shows that symmetry-preserving flows can provide…
by Hoogeboom, Satorras et al. Introduces exponential convolution to add the spatial dependencies in linear layers as an improvement of the 1x1 convolutions. It uses matrix exponentials to create cheap and invertible layers. They also use this new architecture to create convolutional Sylvester…
by Etmann, Ke et al. Extends the classical UNet to be fully invertible by enabling invertible, orthogonal upsampling and downsampling layers. It is rather efficient so it should be able to enable stable training of deeper and larger networks.
by Mahajan, Bhattacharyya et al. Improves the representational power of flow-based models by introducing channel-wise dependencies in their latent space through multi-scale autoregressive priors (mAR). [Code]
by Brehmer, Cranmer Normalizing flows that learn the data manifold and probability density function on that manifold. [Tweet] [Code]
by Meng, Song et al. Uses a repeated composition of trainable kernel layers and orthogonal transformations. Very competitive versus some of the SOTA like Real-NVP, Glow and FFJORD. [Code]
by Giaquinto, Banerjee Augment traditional normalizing flows with gradient boosting. They show that training multiple models can achieve good results and it's not necessary to have more complex distributions. [Code]
by Deng, Chang et al. They propose a normalizing flow using differential deformation of the Wiener process. Applied to time series. [Tweet]
by Hodgkinson, Heide et al. Name clash for a very different technique from the above SNF: an extension of continuous normalizing flows using stochastic differential equations (SDE). Treats Brownian motion in the SDE as a latent variable and approximates it by a flow. Aims to enable efficient…
by Wu, Köhler et al. Introduces SNF, an arbitrary sequence of deterministic invertible functions (the flow) and stochastic processes such as MCMC or Langevin Dynamics. The aim is to increase expressiveness of the chosen deterministic invertible function, while the trainable flow improves sampling…
by Ardizzone, Mackowiak et al. They introduce a class of conditional normalizing flows with an information bottleneck objective. [Code]
by Dolatabadi, Erfani et al. A successor to the Neural spline flows which features an easy-to-compute inverse.
by Papamakarios, Nalisnick et al. A thorough and very readable review article by some of the guys at DeepMind involved in the development of flows. Highly recommended.
by Wehenkel, Louppe UMNN relaxes the constraints on weights and activation functions of monotonic neural networks by setting the derivative of the transformation as the output of an unconstrained neural network. The transformation itself is computed by numerical integration (Clenshaw-Curtis…
by Kobyzev, Prince et al. Another very thorough and very readable review article going through the basics of NFs as well as some of the state-of-the-art. Also highly recommended.
by Rothfuss, Ferreira et al. Normalizing flows for conditional density estimation. This paper proposes noise regularization to reduce overfitting. [Blog]
by Song, Meng et al. Creates an autoregressive-like coupling layer via masked convolutions which is fast and efficient to evaluate. [Code]
by Grcić, Grubišić et al. Creates a nested coupling structure to add more expressivity to standard coupling layers. They also utilize slicing/factorization for dimensionality reduction and Nystromer for the coupling layer conditioning network. They achieved SOTA results for normalizing flow…
by Karami, Schuurmans et al. Introduces convolutional layers that are circular and symmetric. The layer is invertible and cheap to evaluate. They also showcase how one can design non-linear elementwise bijectors that induce special properties via constraining the loss function. [Code]
by Finzi, Izmailov et al. Showcases how standard convolutional layers can be made invertible via Fourier transformations. They also introduce better activations which might be better suited to normalizing flows, e.g. SneakyRELU
by Durkan, Bekasov et al. Uses monotonic ration splines as a coupling layer. This is currently one of the state of the art.
by Liu, Kumar et al. A new, reversible graph network for prediction and generation. They perform similarly to message passing neural networks on supervised tasks, but at significantly reduced memory use, allowing them to scale to larger graphs. Combined with a novel graph auto-encoder for…
by Truong, Luu et al. Seeks to overcome the limitation of 1x1 convolutions and proposes invertible nxn convolutions via a clever convolutional affine function.
by Hoogeboom, Peters et al. A normalizing flow to be used for ordinal discrete data. They introduce a flexible transformation layer called integer discrete coupling.
) by Cao, Titov et al. Introduces (B-NAF), a more efficient probability density approximator. Claims to be competitive with other flows across datasets while using orders of magnitude fewer parameters.
by Ma, Kong et al. Introduces a masked convolutional generative flow (MaCow) layer using a small kernel to capture local connectivity. They showed some improvement over the GLOW model while being fast and stable.
by Hoogeboom, Berg et al. Introduces autoregressive-like convolutional layers that operate on the channel and spatial axes. This improved upon the performance of image datasets compared to the standard 1x1 Convolutions. The trade-off is that the inverse operator is quite expensive however the…
by Kim, Lee et al. A flow-based generative model for raw audio synthesis. [Code]
by Grathwohl, Chen et al. Uses Neural ODEs as a solver to produce continuous-time normalizing flows (CNF).
by Kingma, Dhariwal They show that flows using invertible 1x1 convolution achieve high likelihood on standard generative benchmarks and can efficiently synthesize realistic-looking, large images.
by Inouye, Ravikumar Normalizing flows but from an iterative perspective. Features a Tree-based density estimator.
by Huang, Krueger et al. Unifies and generalize autoregressive and normalizing flow approaches, replacing the (conditionally) affine univariate transformations of MAF/IAF with a more general class of invertible univariate transformations expressed as monotonic neural networks. Also demonstrates…
by Berg, Hasenclever et al. Introduces Sylvester normalizing flows which remove the single-unit bottleneck from planar flows for increased flexibility in the variational posterior.
by Zheng, Yang et al. Introduces normalizing flows that take advantage of convolutions (based on convolution over the dimensions of random input vector) to improve the posterior in the variational inference framework. This also reduced the number of parameters due to the convolutions.
by Papamakarios, Pavlakou et al. Introduces MAF, a stack of autoregressive models forming a normalizing flow suitable for fast density estimation but slow at sampling. Analogous to Inverse Autoregressive Flow (IAF) except the forward and inverse passes are exchanged. Generalization of RNVP.
by Louizos, Welling They introduce a new type of variational Bayesian neural network that uses flows to generate auxiliary random variables which boost the flexibility of the variational family by multiplying the means of a fully-factorized Gaussian posterior over network parameters. This turns…
by Kingma, Salimans et al. Introduces inverse autoregressive flow (IAF), a new type of flow which scales well to high-dimensional latent spaces. [Code]
by Dinh, Sohl-Dickstein et al. They introduce the affine coupling layer (RNVP), a major improvement in terms of flexibility over the additive coupling layer (NICE) with unit Jacobian while keeping a single-pass forward and inverse transformation for fast sampling and density estimation,…
by Rezende, Mohamed They show how to go beyond mean-field variational inference by using flows to increase the flexibility of the variational family.
by Germain, Gregor et al. Introduces MADE, a feed-forward network that uses carefully constructed binary masks on its weights to control the precise flow of information through the network. The masks ensure that each output unit receives signals only from input units that come before it in some…
by Dinh, Krueger et al. Introduces the additive coupling layer (NICE) and shows how to use it for image generation and inpainting.
by Laparra, Camps-Valls et al. Normalizing flows in the form of Gaussianization in an iterative format. Also shows connections to information theory.
by Bézenac, Rangapuram et al. Augments state space models with normalizing flows and thereby mitigates imprecisions stemming from idealized assumptions. Aimed at forecasting real-world data and handling varying levels of missing data. (Also available at Amazon Science.)
by Li, Zhou et al. Proposes to use flows to transform anisotropic sentence embedding distributions from BERT to a smooth and isotropic Gaussian, learned through unsupervised objective. Demonstrates performance gains over SOTA sentence embeddings on semantic textual similarity tasks. Code available…
by Wirnsberger, Ballard et al. Normalizing flows used to estimate free energy differences.
by Siahkoohi, Rizzuti et al. Uses conditional normalizing flows for inverse problems. [Video]
by Lugmayr, Danelljan et al. Uses normalizing flows for super-resolution.
by Hoffman, Sountsov et al. Uses normalizing flows in conjunction with Monte Carlo estimation to have more expressive distributions and better posterior estimation.
by Ardizzone, Kruse et al. Normalizing flows for inverse problems.
by Haarnoja, Hartikainen et al. Uses normalizing flows, specifically RealNVPs, as policies for reinforcement learning and also applies them for the hierarchical reinforcement learning setting.
by Kapil Sachdeva A comprehensive tutorial on flows explaining the challenges addressed by this class of algorithm. Provides intuition on how to address those challenges, and explains the underlying mathematics using a simple step by step approach.
by Marc Deisenroth Part of a NeurIPS 2020 tutorial series titled "There and Back Again: A Tale of Slopes and Expectations". Link to full series.
by Marcus Brubaker A great introduction to normalizing flows by one of the creators of Stan presented at ECCV 2020. The tutorial also provides an excellent review of various practical implementations.
by Pieter Abbeel A really thorough explanation of normalizing flows. Also includes some sample code.
by Ari Seff A great 3blue1brown-style video explaining the basics of normalizing flows.
by Laurent Dinh The first author on both the NICE and RNVP papers and one of the first in this field gives an introductory talk at "Machine Learning for Physics and the Physics Learning of, 2019".
by Jenny Liu Introduces a new graph generating model for use e.g. in drug discovery, where training on molecules that are known to bind/dissolve/etc. may help to generate novel, similarly effective molecules.
by Rianne van den Berg Introduces Sylvester normalizing flows which remove the single-unit bottleneck from planar flows for increased flexibility in the variational posterior.
by François Rozet Zuko is a Python package that implements normalizing flows in PyTorch. It relies heavily on PyTorch's built-in distributions and transformations, which makes the implementation concise, easy to understand and extend. The API is fully documented with references to the original…
by Thorsten Glüsenkamp A package that models joint (conditional) PDFs on tensor products of manifolds (Euclidean, sphere, interval, simplex) - like inverse autoregressive flows, but connects manifolds, models conditional PDFs, and allows for arbitrary couplings instead of affine ones. Includes a…
by Facebook / Meta FlowTorch is a PyTorch library for learning and sampling from complex probability distributions using Normalizing Flows.
by Bayesiains A suite of most of the SOTA methods using PyTorch. From an ML group in Edinburgh. They created the current SOTA spline flows. Almost as complete as you'll find from a single repo.
by Vincent Stimper The library provides most of the common normalizing flow architectures. It also includes stochastic layers, flows on tori and spheres, and other tools that are particularly useful for applications to the physical sciences.
by VLL Heidelberg The Framework for Easily Invertible Architectures (FrEIA) is based on RNVP flows. Easy to setup, it allows to define complex Invertible Neural Networks (INNs) from simple invertible building blocks.
by Google Large first-party library that offers RNVP, MAF among other autoregressive models plus a collection of composable bijectors.
by Meesum Qazalbash, Muhammad Zeeshan et al. A JAX-based gravitational-wave population inference toolkit for parametric models [Docs]
by Kaze Wong Normalizing-flow enhanced sampling package for probabilistic inference [Docs]
by John Franklin Crenshaw A package that focuses on probabilistic modeling of tabular data, with a focus on sampling and posterior calculation.
by DeepMind Distrax is a lightweight library of probability distributions and bijectors. It acts as a JAX-native re-implementation of a subset of TensorFlow Probability (TFP), with some new features and emphasis on extensibility.
by Information Fusion Labs (UMass) A library that offers normalizing flows using JAX as the backend. Has some SOTA methods. They also feature a surjective flow via quantization.
by Hossein Pourbozorg Implementations of Infinitesimal Continuous Normalizing Flows Algorithms in Julia. [Docs]
by SLIM A Flux compatible library implementing invertible neural networks and normalizing flows using memory-efficient backpropagation. Uses manually implemented gradients to take advantage of the invertibility of building blocks, which allows for scaling to large-scale problem sizes.
by Lorenzo Loconte A general-purpose Python library providing a collection of deep probabilistic models (DPMs) which are easy to use and extend. Implements flows such as MAF, RealNVP and NICE.
by Maxime Vandegar PyTorch implementation that reproduces results from the paper NICE in about 100 lines of code.
by pyro.ai A tutorial about how to use the pyro-ppl library (based on PyTorch) to use Normalizing flows. They provide some SOTA methods including NSF and MAF. Parts 2 and 3 coming later.
by torchdyn Example of how to use FFJORD as a continuous normalizing flow (CNF). Based on the PyTorch suite torchdyn which offers continuous neural architectures.
by Rameen Abdal Attribute-conditioned Exploration of StyleGAN-generated Images using Conditional Continuous Normalizing Flows. [Docs]
by Antoine Wehenkel Official implementation of "Graphical Normalizing Flows" and the experiments presented in the paper.
by Andrej Karpathy A Jupyter notebook with PyTorch implementations of the most commonly used flows: NICE, RNVP, MAF, Glow, NSF.
by Antoine Wehenkel Official implementation of "Unconstrained Monotonic Neural Networks" and the experiments presented in the paper.
by acids-ircam A great repo with some basic PyTorch implementations of normalizing flows from scratch.
by Kamen Bliznashki Pytorch implementations of density estimation algorithms: BNAF, Glow, MAF, RealNVP, planar flows.
by Ilya Kostrikov PyTorch implementations of density estimation algorithms: MAF, RNVP, Glow.
by Pierre Segonne This repository provides a hands-on TensorFlow implementation of Normalizing Flows as presented in the paper introducing the concept (D. Rezende & S. Mohamed).
by Lukas Rinder Implementation of normalizing flows (Planar Flow, Radial Flow, Real NVP, Masked Autoregressive Flow (MAF), Inverse Autoregressive Flow (IAF), Neural Spline Flow) in TensorFlow 2 including a small tutorial.
by Bohan Li TensorFlow implementation of "On the Sentence Embeddings from Pre-trained Language Models" (EMNLP 2020).
by numpyro Features an example of how Normalizing flows can be used to get more robust posteriors from Monte Carlo methods. Uses the numpyro library which is a PPL with JAX as the backend. The NF implementations include the basic ones like IAF and BNAF.
by David Inouye Code base for the paper Deep Density Destructors by Inouye & Ravikumar (2018). An entire suite of iterative methods including tree-based as well as Gaussianization methods which are similar to normalizing flows except they converge iteratively instead of fully parametrized. That…
by PyMC3 A very helpful notebook showcasing how to work with flows in practice and comparing it to PyMC3's NUTS-based HMC kernel. Based on Theano.
by Andy Miller Simple didactic example using autograd, so pretty low-level.
by Andrew White A nice introduction starting with the change of variables formula (aka flow equation), going on to cover some common bijectors and finishing with a code example showing how to fit the double-moon distribution with TensorFlow Probability.
by Neal Jean Short and simple explanation of change of variables theorem i.t.o. probability mass conservation.
by Lilian Weng Covers change of variables, NICE, RNVP, MADE, Glow, MAF, IAF, WaveNet, PixelRNN.
by Adam Kosiorek Introduction to flows covering change of variables, planar flow, radial flow, RNVP and autoregressive flows like MAF, IAF and Parallel WaveNet.
by Eric Jang Part 1: Distributions and Determinants. Part 2: Modern Normalizing Flows. Lots of great graphics.
hesreallyhim/awesome-claude-code
A hand-picked collection of the finest of resources for the most awesome of agents, Claude Code, the undisputed champion of coding companions, from the unstoppable team…
VoltAgent/awesome-agent-skills
A curated collection of 1000+ agent skills from official dev teams and the community, compatible with Claude Code, Codex, Gemini CLI, Cursor, and more.
josephmisiti/awesome-machine-learning
A curated list of awesome Machine Learning frameworks, libraries and software.
EthicalML/awesome-production-machine-learning
A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning
academic/awesome-datascience
:memo: An awesome Data Science repository to learn and apply for real world problems.
analysis-tools-dev/static-analysis
⚙️ A curated list of static analysis (SAST) tools and linters for all programming languages, config files, build tools, and more. The focus is on tools which improve…