Affine transformations in Python

Gotchas with Affine Transformations in Python¶

Too-Long, Didn't Read:¶

Use the matplotlib.transforms.Affine2D function to generate transform matrices, and the scipy.ndimage.warp function to warp images using the transform matrices. The skimage AffineTransform shear functionality is weird, and the scipy affine_transform function for warping images swaps the X and Y axes.

Introduction¶

These are several fallpits that I've encountered while learning to use Affine Transformations in Python. There are several libraries that one can use:

For generating the transform matrix

• from matplotlib.transforms import Affine2D
• from skimage.transform import AffineTransform

For warping an image by the transform matrix

• from scipy.ndimage import affine_transform
• from skimage.transform import warp

Fallpit: Which axis is the x-axis?¶

In matplotlib, the x-axis is the second axis of an array. In the example below, we plot an image with shape (768, 1024), and see that the horizontal axis of the image is the longest one. Hence, the shape is (y, x)

Notice also that the origin is in the top left, with the axes increasing right and down. For line plots, the vertical convention is opposite (use ax.invert_yaxis() to compare transforms on points)

In [6]:

from scipy.misc import face

img = face(gray=True)
img.shape

Out[6]:

(768, 1024)

In [7]:

import matplotlib.pyplot as plt
plt.figure()
plt.imshow(img);

This makes sense with how one represents arrays - an array of shape (2,4) is printed in the same orientation as the image.

In [8]:

import numpy as np
A = np.random.random((2,4))
print(A)
plt.figure()
plt.imshow(A);

[[0.51788208 0.94976267 0.69944515 0.4737396 ]
 [0.19868027 0.5998925  0.98593668 0.41154051]]

While this can be a gotcha all on its own, the issue with Affine Transformations arises when we see that the aforementioned packages implement their transformations with different assumptions on what the x-axis should be:

Generating transform matrices¶

We have two main packages for generating transforms such as translation, scale, rotation and shear:

In [9]:

from matplotlib.transforms import Affine2D
from skimage.transform import AffineTransform

First a little aside: The skimage AffineTransform function takes various arguments like scale, shear etc. The matplotlib Affine2D does not do that, but lets you chain methods like Affine2D().scale(2).rotate_deg(45). Both packages have implemented the + operator (via the __add__ method) so that you can chain transforms by adding them together.

Translation¶

In [12]:

Affine2D().translate(tx = 100, ty = 10).get_matrix()

Out[12]:

array([[  1.,   0., 100.],
       [  0.,   1.,  10.],
       [  0.,   0.,   1.]])

In [13]:

AffineTransform(translation=(100, 10))

Out[13]:

<AffineTransform(matrix=
    [[  1.,  -0., 100.],
     [  0.,   1.,  10.],
     [  0.,   0.,   1.]]) at 0x203a7702b50>

Scale¶

In [85]:

Affine2D().scale(sx=2, sy=3).get_matrix()

Out[85]:

array([[2., 0., 0.],
       [0., 3., 0.],
       [0., 0., 1.]])

In [86]:

AffineTransform(scale=(2,3)).params

Out[86]:

array([[ 2., -0.,  0.],
       [ 0.,  3.,  0.],
       [ 0.,  0.,  1.]])

Rotation¶

In [14]:

Affine2D().rotate(np.deg2rad(45)).get_matrix()

Out[14]:

array([[ 0.70710678, -0.70710678,  0.        ],
       [ 0.70710678,  0.70710678,  0.        ],
       [ 0.        ,  0.        ,  1.        ]])

In [15]:

AffineTransform(rotation=np.deg2rad(45)).params

Out[15]:

array([[ 0.70710678, -0.70710678,  0.        ],
       [ 0.70710678,  0.70710678,  0.        ],
       [ 0.        ,  0.        ,  1.        ]])

Shear¶

Here is the first major difference. Both functions take arguments as a shear angle (matplotlib calls the method skew). The matplotlib implementation distinguishes between shear in the two directions, while the skimage only has one. This has been reported as an issue (#3239) in the skimage github repo.

But more concerningly, in the mpl implementation, the same angle, either in x or y, does not give the same answer as the skimage version!

In [43]:

# 45 degree xShear
mplShearX = Affine2D().skew(xShear=np.deg2rad(45), yShear=0).get_matrix()
mplShearX

Out[43]:

array([[1., 1., 0.],
       [0., 1., 0.],
       [0., 0., 1.]])

In [44]:

# 45 degree yShear
mplShearY = Affine2D().skew(xShear=0, yShear=np.deg2rad(45)).get_matrix()
mplShearY

Out[44]:

array([[1., 0., 0.],
       [1., 1., 0.],
       [0., 0., 1.]])

In [45]:

skShear = AffineTransform(shear=np.deg2rad(45)).params
skShear

Out[45]:

array([[ 1.        , -0.70710678,  0.        ],
       [ 0.        ,  0.70710678,  0.        ],
       [ 0.        ,  0.        ,  1.        ]])

In the "Warping functions" section we'll look at what these shearing matrices actually do

Chaining transforms¶

There are several ways of chaining the operators - by using the . operator for the Affine2D() class, by addition of either the Affine2D() or AffineTransform classes or by taking the matrix product on the matrices.

In [106]:

(Affine2D()
    .translate(-200, -100)
    .scale(2, 3)
    .rotate(np.pi/4) 
    .translate(200, 100)
).get_matrix()

Out[106]:

array([[   1.41421356,   -2.12132034,  129.28932188],
       [   1.41421356,    2.12132034, -394.97474683],
       [   0.        ,    0.        ,    1.        ]])

In [107]:

(Affine2D().translate(-200, -100) 
    + Affine2D().scale(2, 3) 
    + Affine2D().rotate(np.pi/4) 
    + Affine2D().translate(200, 100)
).get_matrix()

Out[107]:

array([[   1.41421356,   -2.12132034,  129.28932188],
       [   1.41421356,    2.12132034, -394.97474683],
       [   0.        ,    0.        ,    1.        ]])

In [108]:

(AffineTransform(translation=(-200, -100)) 
    + AffineTransform(scale=(2, 3)) 
    + AffineTransform(rotation=np.pi/4) 
    + AffineTransform(translation=(200, 100)) 
).params

Out[108]:

array([[   1.41421356,   -2.12132034,  129.28932188],
       [   1.41421356,    2.12132034, -394.97474683],
       [   0.        ,    0.        ,    1.        ]])

If one uses the matrix multiplication (dot product) approach, the arrays have to be multiplied in the reverse direction:

In [109]:

(Affine2D().translate(200, 100).get_matrix()
    @ Affine2D().rotate(np.pi/4).get_matrix()
    @ Affine2D().scale(2, 3).get_matrix()
    @ Affine2D().translate(-200, -100).get_matrix()
)

Out[109]:

array([[   1.41421356,   -2.12132034,  129.28932188],
       [   1.41421356,    2.12132034, -394.97474683],
       [   0.        ,    0.        ,    1.        ]])

Warping functions¶

These functions let us apply a transform matrix to an image, so as to warp it.

In [46]:

from skimage.transform import warp
from scipy.ndimage import affine_transform

The first gotcha here is that we have to warp the images by the inverse of the transformation matrix. The inverse matrix maps the positions in the warped array to the original. Personally I find it a bit ridiculous that the package authors haven't used the "forward transform" convention, but it seems that a lot of packages implement it this way.

We do the inverse transform by using numpy's matrix inverse function: T_inv = np.linalg.inv(T).

Let's see what happens if we use the functions to translate an image by 100 in the x-direction:

In [47]:

Tshift = AffineTransform(translation=(100, 0)).params

fig, AX = plt.subplots(ncols=3, figsize=(10,10/3))
(ax0, ax1, ax2) = AX
ax0.imshow(img)
ax1.imshow(warp(img, np.linalg.inv(Tshift)))
ax2.imshow(affine_transform(img, np.linalg.inv(Tshift)))

for ax, title in zip(AX, ['Original', 'skimage warp', 'scipy affine_transform']):
    ax.set_title(title)
plt.tight_layout()

We see here that the warp function from skimage translates in the matplotlib-convention x-direction, while the scipy version translates in the y-direction.

Lets use both functions to see the effect of the three shear matrices from earlier. In the following code, I first write some helper functions to make it easier to plot as a function of angle and implementation

In [79]:

def mpl_shearX(angle):
    return Affine2D().skew(xShear=np.deg2rad(angle), yShear=0).get_matrix()
def mpl_shearY(angle):
    return Affine2D().skew(xShear=0, yShear=np.deg2rad(angle)).get_matrix()
def sk_shear(angle):
    return AffineTransform(shear=np.deg2rad(angle)).params

As we will see, first off the sklearn and scipy warping algorithms work in a mirrored manner, swapping the x and y axes. More interestingly, we see that the skimage shear implementation does not do what is describes - while it does shear along the x-axis, it also seemingly raises the image along the y-axis.

skimage.transform.warp¶

In [80]:

fig, AX = plt.subplots(ncols=4, nrows=3, figsize=(12,9))

fig.suptitle('skimage warp')
angles = [10, 30, 45]
for i, (ax, func) in enumerate(zip(AX, [mpl_shearX, mpl_shearY, sk_shear])):
    (ax0, ax1, ax2, ax3) = ax
    ax0.imshow(img)
    ax1.imshow(warp(img, np.linalg.inv(func(angles[0]))))
    ax2.imshow(warp(img, np.linalg.inv(func(angles[1]))))
    ax3.imshow(warp(img, np.linalg.inv(func(angles[2]))))

AX[0,0].set_title(f"0 deg shear")
AX[0,1].set_title(f"{angles[0]} deg shear")
AX[0,2].set_title(f"{angles[0]} deg shear")
AX[0,3].set_title(f"{angles[0]} deg shear")

AX[0,0].set_ylabel('mpl xShear')
AX[1,0].set_ylabel('mpl yShear')
AX[2,0].set_ylabel('sklearn Shear')

Out[80]:

Text(0, 0.5, 'sklearn Shear')

scipy.ndimage.affine_transform¶

In [82]:

fig, AX = plt.subplots(ncols=4, nrows=3, figsize=(12,9))

fig.suptitle('scipy affine_transform')
angles = [10, 30, 45]
for i, (ax, func) in enumerate(zip(AX, [mpl_shearX, mpl_shearY, sk_shear])):
    (ax0, ax1, ax2, ax3) = ax
    ax0.imshow(img)
    ax1.imshow(affine_transform(img, np.linalg.inv(func(angles[0]))))
    ax2.imshow(affine_transform(img, np.linalg.inv(func(angles[1]))))
    ax3.imshow(affine_transform(img, np.linalg.inv(func(angles[2]))))

AX[0,0].set_title(f"0 deg shear")
AX[0,1].set_title(f"{angles[0]} deg shear")
AX[0,2].set_title(f"{angles[0]} deg shear")
AX[0,3].set_title(f"{angles[0]} deg shear")

AX[0,0].set_ylabel('mpl xShear')
AX[1,0].set_ylabel('mpl yShear')
AX[2,0].set_ylabel('sklearn Shear')

Out[82]:

Text(0, 0.5, 'sklearn Shear')

Conclusions¶

To conclude, I recommend that one uses the matplotlib library to write transforms, and the skimage warp function to warp images. Image transforms can be tricky to get right, and so it helps to be consistent in using the same implementation across all code one writes.

---

<

Previous Post

FFT Speedtest comparing Tensorflow, PyTorch, CuPy, PyFFTW and NumPy

>

Next Post

Test first post