Layer and Activation Functions
We use the following notation to describe layer and activation functions:
Layer Functions
- omlt.neuralnet.layers.full_space.full_space_conv2d_layer(net_block, net, layer_block, layer)[source]
Add full-space formulation of the 2-D convolutional layer to the block
A 2-D convolutional layer applies cross-correlation kernels to a 2-D input. Specifically, the input is convolved by sliding the kernels along the input vertically and horizontally. At each location, the preactivation is computed as the dot product of the kernel weights and the input plus a bias term.
- omlt.neuralnet.layers.full_space.full_space_dense_layer(net_block, net, layer_block, layer)[source]
Add full-space formulation of the dense layer to the block
\[\begin{align*} \hat z_i &= \sum_{j{=}1}^{M_i} w_{ij} z_j + b_i && \forall i \in N \end{align*}\]
- omlt.neuralnet.layers.full_space.full_space_maxpool2d_layer(net_block, net, layer_block, layer)[source]
Add Big-M max pooling formulation.
\[\begin{split}\begin{align*} \hat{z_i} \leq w\cdot x_{i}^{l} + \sum_{k{=}1}^{d} M_{i}^{l,k} q_{i}^{k} && \forall i \in N,\ \forall l \in \{ 1,...,d \} \\ \hat{z_i} \geq w\cdot x_{i}^{l} && \forall i \in N,\ \forall l \in \{ 1,...,d \} \\ (x_{i},\hat{z_i},q_{i}) \in [L_{i},U_{i}] \times \mathbb{R} \times \Delta^{d} && \forall i \in N \\ q_{i} \in \{ 0,1 \}^{d} && \forall i \in N \\ M_{i}^{l,k} = w\cdot max\{ L_{i}^{l} - L_{i}^{k}, \\ L_{i}^{l} - U_{i}^{k}, U_{i}^{l} - L_{i}^{k}, U_{i}^{l} - U_{i}^{k} \} && \forall i \in N,\ \forall l \in \{ 1,...,d \},\ \forall k \in \{ 1,...,d \} \end{align*}\end{split}\]where \(w\) is the convolution kernel on the preceding convolutional layer; \(d\) is the number of features in each of the \(N\) max pooling windows; \(x_{i}\) is the set of \(d\) features in the \(i\)-th max pooling window; \(\Delta^{d}\) is the \(d\)-dimensional simplex; and [L_{i},U_{i}] are the bounds on x_{i}.
NOTE This formulation is adapted from the Anderson et al. (2020) formulation, section 5.1, with the following changes:
OMLT presently does not support biases on convolutional layers. Bias terms from the original formulation are removed.
The original formulation formulates the max of \(w^{l}\cdot x + b^{l}\), varying the weights \(w\) and biases \(b\) and keeping the input \(x\) constant. Since convolutional layers have constant weights and biases convolved with varying portions of the feature map, this formulation formulates the max of \(w\cdot x^{l} + b\).
Due to the above 2 changes, the calculation of \(N^{l,k}\) is changed.
- omlt.neuralnet.layers.reduced_space.reduced_space_dense_layer(net_block, net, layer_block, layer, activation)[source]
Add reduced-space formulation of the dense layer to the block
\[\begin{align*} \hat z_i &= \sum_{j{=}1}^{M_i} w_{ij} z_j + b_i && \forall i \in N \end{align*}\]
- omlt.neuralnet.layers.partition_based.default_partition_split_func(w, n)[source]
Default function for partitioning weights in \(w\) into \(n\) partitions.
Weights in \(w\) are sorted and partitioned evenly.
- omlt.neuralnet.layers.partition_based.partition_based_dense_relu_layer(net_block, net, layer_block, layer, split_func)[source]
Partition-based ReLU activation formulation.
Generates the constraints for the ReLU activation function.
\[\begin{align*} z_i &= \text{max}(0, \hat{z_i}) && \forall i \in N \end{align*}\]The partition-based formulation for the i-th node is given by:
\[\begin{split}\begin{align*} &\sum_{n} \big( \sum_{j \in \mathbb{S}_n} w_{ij} x_j - p_n \big) + \sigma b \leq 0 \\ &\sum_{n} p_n + (1-\sigma) b \geq 0 \\ &z_i = \sum_{n} p_n + (1-\sigma)b \\ &\sigma l_n \leq \sum_{j \in \mathbb{S}_n}w_{ij}x_j - p_n \leq \sigma u_n \\ &(1-\sigma) l_n \leq p_n \leq (1-\sigma) u_n \\ &\sigma \in \{0, 1\} \end{align*}\end{split}\]where \(l_n\) and \(u_n\) are, respectively, lower and upper bounds the n-th partition.
Activation Functions
- omlt.neuralnet.activations.linear.linear_activation_constraint(net_block, net, layer_block, layer, add_constraint=True)[source]
Linear activation constraint generator
Generates the constraints for the linear activation function.
\[\begin{align*} z_i &= \hat{z_i} && \forall i \in N \end{align*}\]
- class omlt.neuralnet.activations.relu.ComplementarityReLUActivation(transform=None)[source]
Bases:
objectComplementarity-based ReLU activation forumlation.
Generates the constraints for the ReLU activation function.
\[\begin{align*} z_i &= \text{max}(0, \hat{z_i}) && \forall i \in N \end{align*}\]The complementarity-based formulation for the i-th node is given by:
\[\begin{align*} 0 &\leq z_i \perp (z-\hat{z_i}) \geq 0 \end{align*}\]
- omlt.neuralnet.activations.relu.bigm_relu_activation_constraint(net_block, net, layer_block, layer)[source]
Big-M ReLU activation formulation.
Generates the constraints for the ReLU activation function.
\[\begin{align*} z_i &= \text{max}(0, \hat{z_i}) && \forall i \in N \end{align*}\]The Big-M formulation for the i-th node is given by:
\[\begin{split}\begin{align*} z_i &\geq \hat{z_i} \\ z_i &\leq \hat{z_i} - l(1-\sigma) \\ z_i &\leq u(\sigma) \\ \sigma &\in \{0, 1\} \end{align*}\end{split}\]where \(l\) and \(u\) are, respectively, lower and upper bounds of \(\hat{z_i}\).
- omlt.neuralnet.activations.smooth.sigmoid_activation_constraint(net_block, net, layer_block, layer)[source]
Sigmoid activation constraint generator
Generates the constraints for the sigmoid activation function.
\[\begin{align*} z_i &= \frac{1}{1 + exp(-\hat z_i)} && \forall i \in N \end{align*}\]
- omlt.neuralnet.activations.smooth.sigmoid_activation_function(x)[source]
Applies the sigmoid function.
\[\begin{align*} y &= \frac{1}{1 + exp(-\hat x)} \end{align*}\]
- omlt.neuralnet.activations.smooth.smooth_monotonic_activation_constraint(net_block, net, layer_block, layer, fcn)[source]
Activation constraint generator for a smooth monotonic function
Generates the constraints for the activation function fcn if it is smooth and monotonic
\[\begin{align*} z_i &= fcn(\hat z_i) && \forall i \in N \end{align*}\]
- omlt.neuralnet.activations.smooth.softplus_activation_constraint(net_block, net, layer_block, layer)[source]
Softplus activation constraint generator
Generates the constraints for the softplus activation function.
\[\begin{align*} z_i &= log(exp(\hat z_i) + 1) && \forall i \in N \end{align*}\]
- omlt.neuralnet.activations.smooth.softplus_activation_function(x)[source]
Applies the softplus function.
\[\begin{align*} y &= log(exp(\hat x) + 1) \end{align*}\]