.. SPDX-FileCopyrightText: 2019-2020 Intel Corporation
..
.. SPDX-License-Identifier: CC-BY-4.0

.. default-domain:: cpp

.. include:: ../replacements.inc.rst

.. _eltwise-link:

###########
Elementwise
###########

The elementwise primitive applies an operation to every element of the tensor.
Variable names follow the standard :ref:`conventions-label`.

.. math:: \dst(\overline{x}) = Operation(\src(\overline{x})),

for :math:`\overline{x} = (x_0, \ldots, x_n)`.

*******
Forward
*******

The following forward operations are supported. Here :math:`s` and :math:`d`
denote :math:`\src` and :math:`\dst`, tensor values respectively.

+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Elementwise algorithm                                  | Forward formula                                                                                                                                             |
+========================================================+=============================================================================================================================================================+
| |eltwise_abs|                                          | :math:`d = \begin{cases} s & \text{if}\ s > 0 \\ -s & \text{if}\ s \leq 0 \end{cases}`                                                                      |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_clip|, |eltwise_clip_use_dst_for_bwd|         | :math:`d = \begin{cases} \beta & \text{if}\ s > \beta \geq \alpha \\ s & \text{if}\ \alpha < s \leq \beta \\ \alpha & \text{if}\ s \leq \alpha \end{cases}` |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_elu|, |eltwise_elu_use_dst_for_bwd|           | :math:`d = \begin{cases} s & \text{if}\ s > 0 \\ \alpha (e^s - 1) & \text{if}\ s \leq 0 \end{cases}`                                                        |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_exp|, |eltwise_exp_use_dst_for_bwd|           | :math:`d = e^s`                                                                                                                                             |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_gelu_erf|                                     | :math:`d = 0.5 s (1 + erf[\frac{s}{\sqrt{2}}])`                                                                                                             |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_gelu_tanh|                                    | :math:`d = 0.5 s (1 + tanh[\sqrt{\frac{2}{\pi}} (s + 0.044715 s^3)])`                                                                                       |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_hardsigmoid|                                  | :math:`d = \text{max}(0, \text{min}(1, \alpha s + \beta))`                                                                                                  |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_hardswish|                                    | :math:`d = s \cdot hardsigmoid(s)`                                                                                                                          |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_linear|                                       | :math:`d = \alpha s + \beta`                                                                                                                                |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_log|                                          | :math:`d = \log_{e}{s}`                                                                                                                                     |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_logistic|, |eltwise_logistic_use_dst_for_bwd| | :math:`d = \frac{1}{1+e^{-s}}`                                                                                                                              |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_mish|                                         | :math:`d = s \cdot \tanh{(\log_{e}(1+e^s))}`                                                                                                                |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_pow|                                          | :math:`d = \alpha s^{\beta}`                                                                                                                                |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_relu|, |eltwise_relu_use_dst_for_bwd|         | :math:`d = \begin{cases} s & \text{if}\ s > 0 \\ \alpha s & \text{if}\ s \leq 0 \end{cases}`                                                                |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_round|                                        | :math:`d = round(s)`                                                                                                                                        |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_soft_relu|                                    | :math:`d =\frac{1}{\alpha} \log_{e}(1+e^{\alpha s})`                                                                                                        |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_sqrt|, |eltwise_sqrt_use_dst_for_bwd|         | :math:`d = \sqrt{s}`                                                                                                                                        |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_square|                                       | :math:`d = s^2`                                                                                                                                             |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_swish|                                        | :math:`d = \frac{s}{1+e^{-\alpha s}}`                                                                                                                       |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_tanh|, |eltwise_tanh_use_dst_for_bwd|         | :math:`d = \tanh{s}`                                                                                                                                        |
+--------------------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+

********
Backward
********

The backward propagation computes :math:`\diffsrc(\overline{s})`, based on
:math:`\diffdst(\overline{s})` and :math:`\src(\overline{s})`. However, some
operations support a computation using :math:`\dst(\overline{s})` memory
produced during forward propagation. Refer to the table above for a list of
operations supporting destination as input memory and the corresponding
formulas.

The following backward operations are supported. Here :math:`s`, :math:`d`,
:math:`ds` and :math:`dd` denote :math:`\src`, :math:`\dst`, :math:`\diffsrc`,
and a :math:`\diffdst` tensor values respectively.

+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Elementwise algorithm              | Backward formula                                                                                                                                                                                                                              |
+====================================+===============================================================================================================================================================================================================================================+
| |eltwise_abs|                      | :math:`ds = \begin{cases} dd & \text{if}\ s > 0 \\ -dd & \text{if}\ s < 0 \\ 0 & \text{if}\ s = 0 \end{cases}`                                                                                                                                |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_clip|                     | :math:`ds = \begin{cases} dd & \text{if}\ \alpha < s < \beta \\ 0 & \text{otherwise}\ \end{cases}`                                                                                                                                            |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_clip_use_dst_for_bwd|     | :math:`ds = \begin{cases} dd & \text{if}\ \alpha < d < \beta \\ 0 & \text{otherwise}\ \end{cases}`                                                                                                                                            |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_elu|                      | :math:`ds = \begin{cases} dd & \text{if}\ s > 0 \\ dd \cdot \alpha e^s & \text{if}\ s \leq 0 \end{cases}`                                                                                                                                     |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_elu_use_dst_for_bwd|      | :math:`ds = \begin{cases} dd & \text{if}\ d > 0 \\ dd \cdot (d + \alpha) & \text{if}\ d \leq 0 \end{cases}` only if :math:`\alpha \geq 0`                                                                                                     |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_exp|                      | :math:`ds = dd \cdot e^s`                                                                                                                                                                                                                     |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_exp_use_dst_for_bwd|      | :math:`ds = dd \cdot d`                                                                                                                                                                                                                       |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_gelu_erf|                 | :math:`ds = dd \cdot \left(0.5 + 0.5 \, \mathrm{erf}\left({\frac{s}{\sqrt{2}}}\right) + \frac{s}{\sqrt{2\pi}}e^{-0.5s^{2}}\right)`                                                                                                            |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_gelu_tanh|                | :math:`\begin{array}{rl} ds = & dd \\ & \cdot 0.5 (1 + \tanh[\sqrt{\frac{2}{\pi}} (s + 0.044715 s^3)]) \\ & \cdot (1 + \sqrt{\frac{2}{\pi}} (s + 0.134145 s^3) \\ & \cdot (1 - \tanh[\sqrt{\frac{2}{\pi}} (s + 0.044715 s^3)]) ) \end{array}` |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_hardsigmoid|              | :math:`ds = \begin{cases} dd \cdot \alpha & \text{if}\ 0 < \alpha s + \beta < 1 \\ 0 & \text{otherwise}\ \end{cases}`                                                                                                                         |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_hardswish|                | :math:`ds = \begin{cases} dd \cdot (2 \alpha + \beta) & \text{if}\ 0 < \alpha s + \beta < 1 \\ dd & \text{if}\ \alpha \cdot s + \beta \geq 1 \\ 0 & \text{otherwise} \end{cases}`                                                             |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_linear|                   | :math:`ds = \alpha \cdot dd`                                                                                                                                                                                                                  |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_log|                      | :math:`ds = \frac{dd}{s}`                                                                                                                                                                                                                     |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_logistic|                 | :math:`ds = \frac{dd}{1+e^{-s}} \cdot (1 - \frac{1}{1+e^{-s}})`                                                                                                                                                                               |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_mish|                     | :math:`ds = dd \cdot \frac{e^{s} \cdot \omega}{\delta^{2}}` with :math:`\omega = e^{3s} + 4 \cdot e^{2s} + e^{s} \cdot (4 \cdot s + 6) + 4 \cdot (s + 1)` and :math:`\delta = e^{2s} + 2 \cdot e^{s} + 2`                                     |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_logistic_use_dst_for_bwd| | :math:`ds = dd \cdot d \cdot (1 - d)`                                                                                                                                                                                                         |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_pow|                      | :math:`ds = dd \cdot \alpha \beta s^{\beta - 1}`                                                                                                                                                                                              |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_relu|                     | :math:`ds = \begin{cases} dd & \text{if}\ s > 0 \\ \alpha \cdot dd & \text{if}\ s \leq 0 \end{cases}`                                                                                                                                         |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_relu_use_dst_for_bwd|     | :math:`ds = \begin{cases} dd & \text{if}\ d > 0 \\ \alpha \cdot dd & \text{if}\ d \leq 0 \end{cases}` only if :math:`alpha \geq 0`                                                                                                            |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_soft_relu|                | :math:`ds = \frac{dd}{1 + e^{-\alpha s}}`                                                                                                                                                                                                     |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_sqrt|                     | :math:`ds = \frac{dd}{2\sqrt{s}}`                                                                                                                                                                                                             |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_sqrt_use_dst_for_bwd|     | :math:`ds = \frac{dd}{2d}`                                                                                                                                                                                                                    |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_square|                   | :math:`ds = dd \cdot 2 s`                                                                                                                                                                                                                     |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_swish|                    | :math:`ds = \frac{dd}{1 + e^{-\alpha s}}(1 + \alpha s (1 - \frac{1}{1 + e^{-\alpha s}}))`                                                                                                                                                     |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_tanh|                     | :math:`ds = dd \cdot (1 - \tanh^2{s})`                                                                                                                                                                                                        |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| |eltwise_tanh_use_dst_for_bwd|     | :math:`ds = dd \cdot (1 - d^2)`                                                                                                                                                                                                               |
+------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

*********************************************************
Difference Between Forward Training and Forward Inference
*********************************************************

There is no difference between the #dnnl_forward_training and
#dnnl_forward_inference propagation kinds.

*******************
Execution Arguments
*******************

When executed, the inputs and outputs should be mapped to an execution
argument index as specified by the following table.

====================== ========================
Primitive input/output Execution argument index
====================== ========================
:math:`\src`           |DNNL_ARG_SRC|
:math:`\dst`           |DNNL_ARG_DST|
:math:`\diffsrc`       |DNNL_ARG_DIFF_SRC|
:math:`\diffdst`       |DNNL_ARG_DIFF_DST|
====================== ========================

*****************
Operation Details
*****************

1. The |eltwise_forward::primitive_desc| and
   |eltwise_backward::primitive_desc| constructors take both
   parameters :math:`\alpha`, and :math:`\beta`. These parameters are
   ignored if they are unused by the algorithm.

2. The memory format and data type for :math:`\src` and :math:`\dst` are
   assumed to be the same. The same holds for :math:`\diffsrc` and :math:`\diffdst`.

3. Both forward and backward propagation support in-place operations, meaning
   that :math:`\src` can be used as input and output for forward propagation,
   and :math:`\diffdst` can be used as input and output for backward
   propagation. In case of an in-place operation, the original data will be
   overwritten. Note, however, that some algorithms for backward propagation
   require original :math:`\src`, hence the corresponding forward propagation
   should not be performed in-place for those algorithms. Algorithms that use
   :math:`\dst` for backward propagation can be safely done in-place.

4. For some operations it might be beneficial to compute backward propagation
   based on :math:`\dst(\overline{s})`, rather than on
   :math:`\src(\overline{s})`, for improved performance.

.. note::

   For operations supporting destination memory as input, :math:`\dst` can be
   used instead of :math:`\src` when backward propagation is computed. This
   enables several performance optimizations (see the tips below).

*****************
Data Type Support
*****************

The eltwise primitive should support the following combinations of data types.

.. note::

   Here we abbreviate data types names for readability. For example, |_f32| is
   abbreviated to |f32|.

================== ==================== ======================
Propagation        Source / Destination Intermediate data type
================== ==================== ======================
forward / backward |f32|, |bf16|        |f32|
forward            |f16|                |f16|
forward            |s32| / |s8| / |u8|  |f32|
================== ==================== ======================

Here the intermediate data type means that the values coming in are first
converted to the intermediate data type, then the operation is applied, and
finally the result is converted to the output data type.

*******************
Data Representation
*******************

The eltwise primitive works with arbitrary data tensors. There is no special
meaning associated with any logical dimensions.

***********************
Post-ops and Attributes
***********************

+-----------+---------------------------------------------------------------------+-------------------------------------------------------------------------------+------------------------+
| Type      | Operation                                                           | Description                                                                   | Restrictions           |
+===========+=====================================================================+===============================================================================+========================+
| Post-op   | :any:`Binary <dnnl::post_ops::append_binary>`                       | Applies a binary operation to the result                                      |                        |
+-----------+---------------------------------------------------------------------+-------------------------------------------------------------------------------+------------------------+

***
API
***

.. doxygenstruct:: dnnl::eltwise_forward
   :project: oneDNN
   :members:

.. doxygenstruct:: dnnl::eltwise_backward
   :project: oneDNN
   :members:


.. vim: ts=3 sw=3 et spell spelllang=en