Vector representations

In NNGeometry, vectors are not just a bunch of scalars, but they have a semantic meaning.

• nngeometry.object.vector.PVector objects are vectors living in the parameter space of a neural network model. An example of such vector is \(\delta \mathbf w\) in the EWC penalty \(\delta \mathbf w^\top F \delta \mathbf w\).

• nngeometry.object.vector.FVector objects are vectors living in the function space of a neural network model. An example of such vector is \(\mathbf{f}=\left(f\left(x_{1}\right),\ldots,f\left(x_{n}\right)\right)^{\top}\) where \(f\) is a neural network and \(x_1,\ldots,x_n\) are examples from a training dataset.

class nngeometry.object.vector.FVector(vector_repr=None)

Bases: object

A vector in function space

to_torch()

class nngeometry.object.vector.PVector(layer_collection, vector_repr=None, dict_repr=None)

Bases: object

A vector in parameter space

Param:

add_to_model(model)

Updates model parameter values by adding the current PVector

Note. This is an inplace operation

clone()

Returns a clone of the current object

copy_to_model(model)

Updates model parameter values with the current PVector

Note. This is an inplace operation

detach()

Detachs the current PVector from the computation graph

dot(other)

Computes the dot product between self and other

Parameters:

other – The other PVector

static from_model(model, layer_collection=None)

Creates a PVector using the current values of the given model

static from_model_grad(model, layer_collection=None)

Creates a PVector using the current values of the .grad fields of parameters of the given model

norm(p=2)

Computes the Lp norm of the PVector

size()

The size of the PVector, or equivalently the number of parameters of the layer collection

to_dict()

to_torch()

Returns a Pytorch 1d tensor of the flatten vector.

Warning

The ordering in which the parameters are flattened can seem to be arbitrary. It is in fact the same ordering as specified by the layercollection.LayerCollection object.

Returns:

a Pytorch Tensor

to_torch_layer(layer_id)

nngeometry.object.vector.random_fvector(n_samples, n_output=1, device=None)

nngeometry.object.vector.random_pvector(layer_collection, device=None, dtype=torch.float64)

Returns a random nngeometry.object.PVector object using the structure defined by the layer_collection parameter, with each components drawn from a normal distribution with mean 0 and standard deviation 1.

The returned PVector will internally use a flat representation.

Parameters:

layer_collection – The nngeometry.layercollection.LayerCollection

describing the structure of the random pvector

nngeometry.object.vector.random_pvector_dict(layer_collection, device=None, dtype=None)

Returns a random nngeometry.object.PVector object using the structure defined by the layer_collection parameter, with each components drawn from a normal distribution with mean 0 and standard deviation 1.

The returned PVector will internally use a dict representation.

Parameters:

layer_collection – The nngeometry.layercollection.LayerCollection

describing the structure of the random pvector