SGHMC

class pysgmcmc.optimizers.sghmc.SGHMC(params, lr: float = 0.01, num_burn_in_steps: int = 3000, noise: float = 0.0, mdecay: float = 0.05, scale_grad: float = 1.0)

Stochastic Gradient Hamiltonian Monte-Carlo Sampler that uses a burn-in procedure to adapt its own hyperparameters during the initial stages of sampling.

See [1] for more details on this burn-in procedure.

See [2] for more details on Stochastic Gradient Hamiltonian Monte-Carlo.

[1] J. T. Springenberg, A. Klein, S. Falkner, F. Hutter

In Advances in Neural Information Processing Systems 29 (2016).

Bayesian Optimization with Robust Bayesian Neural Networks.

[2] T. Chen, E. B. Fox, C. Guestrin

In Proceedings of Machine Learning Research 32 (2014).

Stochastic Gradient Hamiltonian Monte Carlo

__init__(params, lr: float = 0.01, num_burn_in_steps: int = 3000, noise: float = 0.0, mdecay: float = 0.05, scale_grad: float = 1.0) → None

Set up a SGHMC Optimizer.

Parameters:

• params (iterable) – Parameters serving as optimization variable.
• lr (float, optional) – Base learning rate for this optimizer. Must be tuned to the specific function being minimized. Default: 1e-2.
• num_burn_in_steps (int, optional) – Number of burn-in steps to perform. In each burn-in step, this sampler will adapt its own internal parameters to decrease its error. Set to 0 to turn scale adaption off. Default: 3000.
• noise (float, optional) – (Constant) per-parameter noise level. Default: 0..
• mdecay (float, optional) – (Constant) momentum decay per time-step. Default: 0.05.
• scale_grad (float, optional) – Value that is used to scale the magnitude of the noise used during sampling. In a typical batches-of-data setting this usually corresponds to the number of examples in the entire dataset. Default: 1.0.

step(closure=None)

Performs a single optimization step (parameter update).

Parameters:closure (callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.