ls_mlkit.optimizer.kfa package

Submodules

• ls_mlkit.optimizer.kfa.kfa module
• ls_mlkit.optimizer.kfa.kfa_optimizer module

Module contents

class ls_mlkit.optimizer.kfa.KFA(model)

Bases: object

static calculate_fisher_inverse_mult_V(cache: dict, a: Tensor, g: Tensor, V: Tensor)

V:(…, m, n) aaT:(…, m, m) ggT:(…, n, n)

eps = 0.001

static get_save_hook_for_a(cache: dict, module_dot_path: str, name: str = 'a')

static get_save_hook_for_g(cache: dict, module_dot_path: str, name: str = 'g')

known_modules = ['Linear']

register_save_hook(model)

remove_hook()

class ls_mlkit.optimizer.kfa.KFAOptimizer(params, base_optimizer: Optimizer, model: Module, user_config: UserConfig, kfa: KFA, **kwargs)

Bases: Optimizer

back_to(name)

calculate_C_inverse_mult_d(tgt_key: str = 'inv_Cd')

calculate_fisher_inverse_mult(tgt_key: str, src_key: str)

epsilon_perturb()

get_something_norm(something_name: Literal['grad', 'state', 'weight'] = 'grad', **kwargs)

load_state_dict(state_dict)

Load the optimizer state.

Parameters:

state_dict (dict) – optimizer state. Should be an object returned from a call to state_dict().

Warning

Make sure this method is called after initializing torch.optim.lr_scheduler.LRScheduler, as calling it beforehand will overwrite the loaded learning rates.

Note

The names of the parameters (if they exist under the “param_names” key of each param group in state_dict()) will not affect the loading process. To use the parameters’ names for custom cases (such as when the parameters in the loaded state dict differ from those initialized in the optimizer), a custom register_load_state_dict_pre_hook should be implemented to adapt the loaded dict accordingly. If param_names exist in loaded state dict param_groups they will be saved and override the current names, if present, in the optimizer state. If they do not exist in loaded state dict, the optimizer param_names will remain unchanged.

Example

>>> # xdoctest: +SKIP
>>> model = torch.nn.Linear(10, 10)
>>> optim = torch.optim.SGD(model.parameters(), lr=3e-4)
>>> scheduler1 = torch.optim.lr_scheduler.LinearLR(
...     optim,
...     start_factor=0.1,
...     end_factor=1,
...     total_iters=20,
... )
>>> scheduler2 = torch.optim.lr_scheduler.CosineAnnealingLR(
...     optim,
...     T_max=80,
...     eta_min=3e-5,
... )
>>> lr = torch.optim.lr_scheduler.SequentialLR(
...     optim,
...     schedulers=[scheduler1, scheduler2],
...     milestones=[20],
... )
>>> lr.load_state_dict(torch.load("./save_seq.pt"))
>>> # now load the optimizer checkpoint after loading the LRScheduler
>>> optim.load_state_dict(torch.load("./save_optim.pt"))

perturb(name: Literal['grad', 'state'] = 'grad', **kwargs)

save_grad(name: str)

save_moving_average_and_save_d(name: str, alpha: float = 0.9)

save_params(name: str)

set_closure(loss_fn, x, y, closure=None)

state_dict()

Return the state of the optimizer as a dict.

It contains two entries:

• state: a Dict holding current optimization state. Its content

  differs between optimizer classes, but some common characteristics hold. For example, state is saved per parameter, and the parameter itself is NOT saved. state is a Dictionary mapping parameter ids to a Dict with state corresponding to each parameter.

• param_groups: a List containing all parameter groups where each

  parameter group is a Dict. Each parameter group contains metadata specific to the optimizer, such as learning rate and weight decay, as well as a List of parameter IDs of the parameters in the group. If a param group was initialized with named_parameters() the names content will also be saved in the state dict.

NOTE: The parameter IDs may look like indices but they are just IDs associating state with param_group. When loading from a state_dict, the optimizer will zip the param_group params (int IDs) and the optimizer param_groups (actual nn.Parameter s) in order to match state WITHOUT additional verification.

A returned state dict might look something like:

{
    'state': {
        0: {'momentum_buffer': tensor(...), ...},
        1: {'momentum_buffer': tensor(...), ...},
        2: {'momentum_buffer': tensor(...), ...},
        3: {'momentum_buffer': tensor(...), ...}
    },
    'param_groups': [
        {
            'lr': 0.01,
            'weight_decay': 0,
            ...
            'params': [0]
            'param_names' ['param0']  (optional)
        },
        {
            'lr': 0.001,
            'weight_decay': 0.5,
            ...
            'params': [1, 2, 3]
            'param_names': ['param1', 'layer.weight', 'layer.bias'] (optional)
        }
    ]
}

step(closure=None)

Perform a single optimization step to update parameter.

Parameters:

closure (Callable) – A closure that reevaluates the model and returns the loss. Optional for most optimizers.

zero_grad()

Reset the gradients of all optimized torch.Tensor s.

Parameters:

set_to_none (bool, optional) –

Instead of setting to zero, set the grads to None. Default: True

This will in general have lower memory footprint, and can modestly improve performance. However, it changes certain behaviors. For example:

1. When the user tries to access a gradient and perform manual ops on it, a None attribute or a Tensor full of 0s will behave differently.

2. If the user requests zero_grad(set_to_none=True) followed by a backward pass, .grads are guaranteed to be None for params that did not receive a gradient.

3. torch.optim optimizers have a different behavior if the gradient is 0 or None (in one case it does the step with a gradient of 0 and in the other it skips the step altogether).

class ls_mlkit.optimizer.kfa.UserConfig(xi: float = 0.0001, alpha: float = 0.9, rho: float = 0.1, rho_cov: float = 0.1)

Bases: object

alpha: float = 0.9

rho: float = 0.1

rho_cov: float = 0.1

xi: float = 0.0001