# LeakyReLU#

LeakyReLU operation is a type of activation function based on ReLU. It has a small slope for negative values with which LeakyReLU can produce small, non-zero, and constant gradients with respect to the negative values. The slope is also called the coefficient of leakage.

Unlike PReLU, the coefficient $$\alpha$$ is constant and defined before training.

LeakyReLU operation applies following formula on every element of src tensor (the variable names follow the standard Conventions):

$\begin{split}\dst = \begin{cases} \src & \text{if}\ \src \ge 0 \\ \alpha \src & \text{if}\ \src < 0 \end{cases}\end{split}$

## Operation Attributes#

Attribute

Name

Description

Value Type

Supported

Values

Required or

Optional

alpha

Alpha is the coefficient of leakage.

f32

Arbitrary f32 value but usually a small positive value.

Required

## Execution Arguments#

The inputs and outputs must be provided according to the below index order when constructing an operation.

### Inputs#

Index

Argument Name

Required or Optional

0

src

Required

### Outputs#

Index

Argument Name

Required or Optional

0

dst

Required

## Supported Data Types#

LeakyReLU operation supports the following data type combinations.

Src

Dst

f32

f32

bf16

bf16

f16

f16