The Smooth Edge feature is an artificial method to reduce aliasing artifacts at the transformed source image edges. Aliasing artifacts may appear because the transformation algorithms skip a destination pixel if its source origin is out of the source image ROI. Thus, borders of the transformed source image can look stepped:
If the smoothEdge flag is set, destination pixels that are closest to the transformed source image edges are mixed with sampled source pixels by the following formula:
srcSampled is the intensity of the source pixel after transformation.
dstExist is the intensity of the destination pixel before transformation.
a is the weight of the outer pixel; set by the function.
dstRes is the intensity of the resulting destination pixel.
The edge smoothing method is not universal: in some cases it can improve the image, but in other cases it can be inefficient. For example, edge smoothing does not increase the quality of images with high contrast borders, and it is not recommended to apply edge smoothing to such images.
Edge smoothing is a post-processing operation: it is performed after transformation. When warping a tiled image, artifacts may appear on tile borders. In this case, edges are not smoothed.