{"id":10198,"date":"2020-02-07T00:05:00","date_gmt":"2020-02-06T21:05:00","guid":{"rendered":"https:\/\/handoli.com\/index.php\/2020\/02\/07\/upsampling-in-core-ml\/"},"modified":"2020-02-07T00:05:00","modified_gmt":"2020-02-06T21:05:00","slug":"upsampling-in-core-ml","status":"publish","type":"post","link":"https:\/\/handoli.com\/index.php\/2020\/02\/07\/upsampling-in-core-ml\/","title":{"rendered":"Upsampling in Core ML"},"content":{"rendered":"<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/Bilinear@2x.jpg\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/Bilinear.jpg\" alt=\"Upsampling a feature map\"\/><\/a><br \/>\n<\/figure>\n<p>Resizing feature maps is a common operation in many neural networks, especially those that perform some kind of image segmentation task.<\/p>\n<p>One issue I ran into recently while converting a neural network to Core ML, is that the original PyTorch model gave <strong>different results<\/strong> for its bilinear upsampling than Core ML, and I wanted to understand why.<\/p>\n<p>When converting models between deep learning frameworks, it\u2019s common to find small differences between the original model and the converted one, as layers aren\u2019t necessarily implemented in exactly the same way in every framework.<\/p>\n<p>This is not always a problem. Most of the time, even if the results of upsampling are off by a few pixels, the model will still work correctly.<\/p>\n<p>However, such differences do make it tricky to verify that the conversion was 100% successful, which is especially important if you have to do post-processing on the model\u2019s predictions afterwards and you want to make sure the output is correct by comparing it to the original model.<\/p>\n<p>So I wanted to find out what exactly was going on\u2026<\/p>\n<div class=\"note\">\n<p><strong>Tip:<\/strong> At the end of this post there is a handy cheatsheet that lets you look up which Core ML layer to use with your TensorFlow \/ Keras \/ PyTorch models.<\/p>\n<\/div>\n<h2>Core ML has multiple ways to resize data<\/h2>\n<p>There are several ways to do <strong>upsampling<\/strong> in Core ML:<\/p>\n<ul>\n<li><code>Upsample<\/code> layer<\/li>\n<li><code>ResizeBilinear<\/code> layer<\/li>\n<li><code>CropResize<\/code> layer<\/li>\n<li>deconvolution<\/li>\n<li>\u201cPixel Shuffle\u201d<\/li>\n<li>using a custom layer<\/li>\n<\/ul>\n<p><code>Upsample<\/code> and <code>ResizeBilinear<\/code> appear to do roughly the same thing, but there are some differences:<\/p>\n<ul>\n<li><code>Upsample<\/code> has <strong>nearest neighbor<\/strong> and <strong>bilinear<\/strong> modes, while <code>ResizeBilinear<\/code> can only do bilinear (the name kind of gives it away).<\/li>\n<li><code>Upsample<\/code> takes an <strong>integer scaling factor<\/strong> (2\u00d7, 3\u00d7, etc). It can only do upscaling, not downscaling.<\/li>\n<li><code>ResizeBilinear<\/code> takes a target <strong>width and height<\/strong>. This may be smaller than the input, so you can also use it to resize the feature maps down.<\/li>\n<li>The <code>CropResize<\/code> layer is like <code>ResizeBilinear<\/code> but also lets you crop out one or more rectangles (ROIs). I\u2019m not going to talk about it in this blog post; this layer works the same way as <code>ResizeBilinear<\/code> when it comes to resizing.<\/li>\n<\/ul>\n<p>The biggest difference between these layers and an API such as Core Image\u2019s <code>CILanczosScaleTransform<\/code> or Accelerate\u2019s <code>vImageScale<\/code>, is that they work on feature maps that may have many more channels than the 3 or 4 channels in a regular RGBA image. Other than that, they do pretty much the same thing.<\/p>\n<p>This blog post is mostly about upsampling, but convolutional neural networks also have various ways to <strong>downsample<\/strong> feature maps. This is typically done using a conv layer with stride 2 or using pooling layers. And of course, <code>ResizeBilinear<\/code> can also scale down.<\/p>\n<h3>Sampling modes<\/h3>\n<p>The <code>Upsample<\/code> layer doesn\u2019t have many options, it only lets you choose between NN (nearest neighbor) and bilinear interpolation.<\/p>\n<p><code>ResizeBilinear<\/code> doesn\u2019t have an option for nearest neighbors, but it does allow you to select the sampling mode that will be used by the bilinear interpolation. This sampling mode is actually very relevant to our investigation.<\/p>\n<p>The possible sampling modes are:<\/p>\n<ul>\n<li><strong>STRICT_ALIGN_ENDPOINTS_MODE<\/strong><\/li>\n<li><strong>ALIGN_ENDPOINTS_MODE<\/strong><\/li>\n<li><strong>UPSAMPLE_MODE<\/strong> \u2014 gives same results as <code>Upsample<\/code> layer<\/li>\n<li><strong>ROI_ALIGN_MODE<\/strong><\/li>\n<\/ul>\n<p>The difference between these sampling modes is in how they determine which pixels to read from the source tensor. The two modes we\u2019re going to look at in this blog post are <code>STRICT_ALIGN_ENDPOINTS_MODE<\/code> and <code>UPSAMPLE_MODE<\/code>.<\/p>\n<p>When you use <code>UPSAMPLE_MODE<\/code> with a <code>ResizeBilinear<\/code> layer, it gives exactly the same results as the <code>Upsample<\/code> layer in bilinear mode. That means you can use <code>Upsample<\/code> if you have an integer scaling factor and <code>ResizeBilinear<\/code> if you don\u2019t.<\/p>\n<p>What about <code>ALIGN_ENDPOINTS_MODE<\/code>? Well, there is no real difference between <code>STRICT_ALIGN_ENDPOINTS_MODE<\/code> and <code>ALIGN_ENDPOINTS_MODE<\/code>, except when the output tensor is just one pixel wide (or tall). In \u201cstrict\u201d align mode, this would sample from pixel 0 in the source image, but in <code>ALIGN_ENDPOINTS_MODE<\/code> it samples from the center pixel. That\u2019s the only place they differ \u2014 in any other situations, both modes work the same.<\/p>\n<p>We\u2019ll conveniently skip over <code>ROI_ALIGN_MODE<\/code> as I don\u2019t really know what it\u2019s used for (but I suspect it\u2019s intended for the <code>CropResize<\/code> layer.)<\/p>\n<p>OK, but what do these sampling modes actually <em>do<\/em>? I\u2019ll show you in moment what <code>STRICT_ALIGN_ENDPOINTS_MODE<\/code> and <code>UPSAMPLE_MODE<\/code> look like, and when you\u2019d use these in your model.<\/p>\n<h3>Pixel Shuffle<\/h3>\n<p>An interesting way to upsample data is to use \u201cpixel shuffle\u201d. This can avoid pixel artifacts that may be introduced by other methods, in particular by deconvolution.<\/p>\n<p>Let\u2019s say you want to upsample a feature map with dimensions H\u00d7W by a factor 2. Pixel shuffle groups the feature map\u2019s channels into sets of 4 and then reorganizes each group into a 2\u00d72 block of pixels. The output feature map will have size 2H\u00d72W but only 1\/4th the number of channels. It\u2019s the same data, just reorganized.<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/PixelShuffle@2x.jpg\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/PixelShuffle.jpg\" alt=\"The pixel shuffle operation\"\/><\/a><br \/>\n<\/figure>\n<div class=\"image-credit\">\n<p>Illustration from the paper <em>Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network<\/em> by Shi et al, 2016.<\/p>\n<\/div>\n<p>Pixel shuffle is not a built-in Core ML operation but is still possible. As this is really just a combination of reshape and transpose operations, you can implement pixel shuffle using a combination of different Core ML <code>Reshape<\/code> and <code>Permute<\/code> layers.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> I might write a future blog post on how to do pixel shuffle with Core ML. In the mean time, if you\u2019re curious, check out how <a href=\"https:\/\/github.com\/onnx\/onnx-coreml\/blob\/7f7a30cae66e7d65917ceea966225292f582d809\/onnx_coreml\/_transformers.py#L418\">onnx_coreml<\/a> does it.<\/p>\n<\/div>\n<h3>What doesn\u2019t Core ML do?<\/h3>\n<p>There is no <code>NearestNeighborResize<\/code> in Core ML currently, so if you want to use \u201cnearest\u201d resizing, you\u2019ll have to use an <code>Upsample<\/code> layer.<\/p>\n<p>The layers provided by Core ML only do <strong>nearest neighbor<\/strong> and <strong>bilinear<\/strong> scaling. Other common interpolation methods are <strong>bicubic<\/strong> and <strong>Lanczos<\/strong> scaling, but these are not supported directly by Core ML.<\/p>\n<p>The biggest issue for me, however, is that for both <code>Upsample<\/code> and <code>ResizeBilinear<\/code> layers, the scaling factor \/ target size are <strong>hardcoded in the model<\/strong>. They cannot be provided by another tensor.<\/p>\n<p>This might be an issue if your model needs to work with multiple input sizes, i.e. if it uses Core ML\u2019s \u201cflexibility\u201d feature:<\/p>\n<ul>\n<li>If you need to resize the feature maps proportionally to the size of the input, you can only use <code>Upsample<\/code>.<\/li>\n<li>Unfortunately, <code>ResizeBilinear<\/code> is only useful if you already know what the output size will be ahead of time.<\/li>\n<\/ul>\n<p>On the other hand, <code>Upsample<\/code> cannot resize by arbitrary amounts, only 1\u00d7, 2\u00d7, 3\u00d7, and so on. <code>ResizeBilinear<\/code> can scale up to any size.<\/p>\n<p>These restrictions put a serious limit on how flexible you can be in your models\u2026<\/p>\n<p>And it\u2019s important to realize what these limitations are, <strong>before you start training your model<\/strong> (ergo, this blog post).<\/p>\n<p>It would be nice if Core ML had a general-purpose \u201cResize\u201d layer that can scale to any size, or by any fractional scaling factor, using a variety of interpolation methods. And that these properties can be either hardcoded into the mlmodel file or be provided dynamically by the output of another layer. (Maybe in Core ML 4?)<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> Of course, if your model needs a method of upsampling that isn\u2019t covered by Core ML\u2019s built-in operations, you can always <a href=\"https:\/\/machinethink.net\/blog\/coreml-custom-layers\/\">make a custom layer<\/a>.<\/p>\n<\/div>\n<p>OK, let\u2019s have a look now at what Core ML <em>does<\/em> support.<\/p>\n<h2>Nearest neighbors<\/h2>\n<p>Nearest neighbors, or NN for short, is a very simple method that basically makes everything more blocky:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/Nearest@2x.jpg\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/Nearest.jpg\" alt=\"Nearest neighbor interpolation\"\/><\/a><br \/>\n<\/figure>\n<p>In Core ML you\u2019ll have to use the <code>Upsample<\/code> layer for this, in mode <code>\"NN\"<\/code>.<\/p>\n<p>NN scaling usually doesn\u2019t give any problems: all ML frameworks implement this in pretty much the same way.<\/p>\n<p>The only limitation of Core ML is that you always need to use an whole number as the scale factor. Other frameworks usually don\u2019t have this limitation.<\/p>\n<p>If you\u2019re using a model that has NN upsampling but with a fractional scaling factor, the Core ML conversion may succeed but the <code>Upsample<\/code> layer may have <strong>the wrong scaling factor<\/strong>! So make sure to check your mlmodel file with <a href=\"https:\/\/github.com\/lutzroeder\/Netron\">Netron<\/a> afterwards.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> If your model has an NN upsample layer that does not have a scaling factor but specifies a width and height, the Core ML converter may still use an <code>Upsample<\/code> layer and translate the width and height into scaling factors. Again, check the mlmodel file to make sure this was actually done correctly.<\/p>\n<\/div>\n<p>Of course, the downside of NN scaling is that the results don\u2019t look particularly nice, so you typically want to use bilinear scaling. And that\u2019s where the trouble starts\u2026<\/p>\n<h2>Bilinear resizing<\/h2>\n<p>When you upsample, you go from a smaller image \u2014 or tensor in this case \u2014 to a larger one that has more pixels. The question is: how will you fill up those extra pixels?<\/p>\n<p>With nearest neighbors you simply duplicate the existing pixel values, but that gives blocky results.<\/p>\n<p>Bilinear interpolation is smarter: to figure out what value a pixel in the destination image will have, you take that pixel\u2019s x,y-coordinate and map it to back a position in the source image.<\/p>\n<p>Often you\u2019ll end up at a fractional coordinate, somewhere in between two pixels. Nearest neighbors simply rounds off that coordinate to the nearest integer. But bilinear interpolation looks at the surrounding 2\u00d72 pixels and figures out how much each of these pixels should contribute.<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/Algorithm@2x.png\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/Algorithm.png\" alt=\"Mapping the destination coordinate to the source image\"\/><\/a><br \/>\n<\/figure>\n<p>The closer our fractional subpixel coordinate is to one of these neighbor pixels, the more that pixel counts. For example, if the coordinate is right in the middle of the group of 2\u00d72 pixels, they all contribute 25%.<\/p>\n<p>The final pixel value is therefore a <strong>weighted average<\/strong> of the values of these four source pixels.<\/p>\n<p>Just in case you\u2019re curious, here is a simple Python routine that implements this algorithm:<\/p>\n<pre><code class=\"language-python\">def resize_bilinear(input, scale_factor_h, scale_factor_w):\n    source_H = input.shape[0]\n    source_W = input.shape[1]\n\n    resized_H = int(source_H * scale_factor_h)\n    resized_W = int(source_W * scale_factor_w)\n\n    output = np.zeros((resized_H, resized_W), dtype=np.float32)\n    \n    def read_pixel(x, y):\n        x = np.clip(x, 0, source_W - 1)\n        y = np.clip(y, 0, source_H - 1)\n        return input[y, x]\n\n    def bilinear_interpolate(x, y):\n        x1 = int(np.floor(x))\n        x2 = x1 + 1\n\n        y1 = int(np.floor(y))\n        y2 = y1 + 1\n\n        P11 = read_pixel(x1, y1)\n        P12 = read_pixel(x1, y2)\n        P21 = read_pixel(x2, y1)\n        P22 = read_pixel(x2, y2)\n        \n        return (P11 * (x2 - x) * (y2 - y) + \n                P12 * (x2 - x) * (y - y1) + \n                P21 * (x - x1) * (y2 - y) + \n                P22 * (x - x1) * (y - y1)) \/ ((x2 - x1) * (y2 - y1))\n\n    for dst_y in range(resized_H):\n        for dst_x in range(resized_W):\n            src_x = (dst_x + 0.5) \/ scale_factor_w - 0.5\n            src_y = (dst_y + 0.5) \/ scale_factor_h - 0.5\n            output[dst_y, dst_x] = bilinear_interpolate(src_x, src_y)\n\n    return output\n<\/code><\/pre>\n<p>I used the following code to create the input image shown in the examples. It has width 10 and height 9:<\/p>\n<pre><code class=\"language-python\">np.random.seed(12345)\nfeature_map = np.random.randn(9, 10).astype(np.float32)\n<\/code><\/pre>\n<p>After bilinearly resizing it to twice the width and height, this is the result:<\/p>\n<pre><code class=\"language-python\">output = resize_bilinear(feature_map, 2, 2)\nplt.imshow(output)\n<\/code><\/pre>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/PythonOutput@2x.png\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/PythonOutput.png\" alt=\"Is that a wig?\"\/><\/a><br \/>\n<\/figure>\n<p>You\u2019d think that everyone would implement bilinear resizing in the exact same way, but that would be too easy\u2026 There are basically three different methods in use by today\u2019s machine learning frameworks:<\/p>\n<ol>\n<li>the bad method<\/li>\n<li>the so-so method<\/li>\n<li>the good method<\/li>\n<\/ol>\n<p>Let\u2019 s take a look at each of these in turn, and figure out where Core ML fits into this picture.<\/p>\n<h2>The bad method<\/h2>\n<p>You can blame TensorFlow for this one. The original bilinear image resizing function that was built into <strong>TensorFlow 1.x<\/strong> is incorrect. For backward compatibility reasons, this was not fixed and it has confused machine learning practitioners ever since.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> The discussion in this section only applies to TensorFlow 1.x and Keras. It does not apply to PyTorch or TensorFlow 2.<\/p>\n<\/div>\n<p>When you use <code>tf.image.resize_bilinear(image, align_corners=False)<\/code> or <code>tf.image.resize_images(image, method=BILINEAR, align_corners=False)<\/code>, the output looks like this:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/Wrong@2x.jpg\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/Wrong.jpg\" alt=\"The wrong method of bilinear upsampling\"\/><\/a><br \/>\n<\/figure>\n<p>Why is this bad? One obvious place is the last row and column: you can clearly see that the pixels are duplicated here. But there\u2019s more going on. If you compare it to the picture from the previous section, you can see that it\u2019s much \u201cnarrower\u201d.<\/p>\n<p>Here is an animated GIF that flips between the NN version and the TensorFlow upscaling:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/NN+Wrong.gif\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/NN+Wrong.gif\" alt=\"Animation between nearest neighbors and the wrong method\"\/><\/a><br \/>\n<\/figure>\n<p>As you can see, the pixels shift up and to the left. That\u2019s not supposed to happen when you resize an image\u2026<\/p>\n<p>Bad or not, if you have a TensorFlow 1.x model that uses this method, <strong>you\u2019ll want Core ML to do the same thing<\/strong>. This is also true for Keras models using an <code>UpSampling2D<\/code> layer running on the TF 1.x backend (but not for Keras on TF 2.x, more about this shortly).<\/p>\n<p>Well, you\u2019re in luck! This is also the result from Core ML\u2019s <code>Upsample<\/code> layer when used in <code>\"BILINEAR\"<\/code> mode and of the <code>ResizeBilinear<\/code> layer in <code>UPSAMPLE_MODE<\/code>.<\/p>\n<p>It makes sense that the creators of Core ML wanted it to be compatible with TensorFlow, as that is still the world\u2019s most popular deep learning framework. Unfortunately, this method of bilinear resizing is not compatible with pretty much any other ML framework or image library.<\/p>\n<p>So if you\u2019re using the <code>Upsample<\/code> layer in your Core ML model \u2014 or <code>ResizeBilinear<\/code> in <code>UPSAMPLE_MODE<\/code> \u2014 you\u2019re getting the behavior from TensorFlow 1.x\u2019s incorrect method. That\u2019s perfectly fine if your model was trained with TF 1.x or Keras, but if your model was not made with TensorFlow, this might not be optimal!<\/p>\n<p>The creators of TensorFlow realized their mistake and but it took two attempts to fix. Let\u2019s look at the first fix now.<\/p>\n<h2>The so-so method<\/h2>\n<p>To fix the issues with the bad method, the <code>align_corners=True<\/code> parameter was introduced. For compatibility reasons this parameter defaults to <code>False<\/code>, so unless your code is explicitly setting it to <code>True<\/code>, you\u2019re still using the old, wrong method!<\/p>\n<p>When you call <code>tf.image.resize_bilinear(image, align_corners=True)<\/code>, this is what the result is like:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/AlignCorners@2x.jpg\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/AlignCorners.jpg\" alt=\"The align_corner=True method of bilinear upsampling\"\/><\/a><br \/>\n<\/figure>\n<p>That looks better than the wrong method indeed.<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/NN+AlignCorners.gif\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/NN+AlignCorners.gif\" alt=\"Animation between nearest neighbors and the align-corners method\"\/><\/a><br \/>\n<\/figure>\n<p>With <code>align_corners=True<\/code>, it preserves the corner points of the input tensor. Sounds like a good thing, but the consequence is that this new method is not very consistent in which pixels it reads from the input tensor. Let\u2019s look at an example.<\/p>\n<p>In the illustrations for this blog post I\u2019m using a tensor of width 10 and height 9. (The number of channels doesn\u2019t matter, as each channel is resized independently of the others). We\u2019re upsampling this by a factor of 2\u00d7, to a new tensor of width 20 and height 18.<\/p>\n<p>Looking just at the x-coordinates, for bilinear resizing with <code>align_corners=True<\/code>, we will sample pixels at the following x-coordinates from the source tensor:<\/p>\n<pre><code class=\"language-nohighlight\">0.0,        0.47368421, 0.94736842, 1.42105263, 1.89473684,\n2.36842105, 2.84210526, 3.31578947, 3.78947368, 4.26315789,\n4.73684211, 5.21052632, 5.68421053, 6.15789474, 6.63157895,\n7.10526316, 7.57894737, 8.05263158, 8.52631579, 9.0\n<\/code><\/pre>\n<p>Note that there are 20 values here, spaced evenly from 0 (the first pixel on the left) to 9 (the last pixel on the right). These two corner points are whole numbers. This is what is meant by \u201cpreserves the corner points\u201d.<\/p>\n<p>The other coordinates are not whole numbers, meaning that they fall somewhere in between two pixels, but that\u2019s OK \u2014 we will apply the bilinear interpolation formula to read the 2\u00d72 neighboring pixels and compute a weighted average of their values.<\/p>\n<p>The reason why <code>align_corners=True<\/code> still isn\u2019t a great solution, is that we\u2019d expect the sampling coordinates to always be the same for the same scaling factor. But if you change the size of the input tensor, the sampling coordinates also change with this method.<\/p>\n<p>For example, let\u2019s say we have a tensor of width 20 and we want to resize by 2\u00d7. The x-coordinates are now:<\/p>\n<pre><code class=\"language-nohighlight\">0.0,         0.48717949,  0.97435897,  1.46153846,  1.94871795,\n2.43589744,  2.92307692,  3.41025641,  3.89743590,  4.38461538,\n4.87179487,  5.35897436,  5.84615385,  6.33333333,  6.82051282,\n7.30769231,  7.79487179,  8.28205128,  8.76923077,  9.25641026\n...\n17.05128205, 17.53846154, 18.02564103, 18.51282051, 19.0  \n<\/code><\/pre>\n<p>I didn\u2019t show all 40 numbers, but you can clearly see that the sample coordinates are slightly different. This makes the distance between successive pixels depend on the size of the image \u2014 which is strange because you\u2019d think pixels are always the same distance apart. Not ideal.<\/p>\n<h3>How does this compare to align_corners=False?<\/h3>\n<p>A quick example to illustrate the difference with TensorFlow\u2019s \u201cbad\u201d method, i.e. when <code>align_corners=False<\/code>. To upsample the width 10 tensor, we\u2019d read from the source tensor at the following x-coordinates:<\/p>\n<pre><code class=\"language-nohighlight\">0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, \n5.0, 5.5, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5, 9.0, 9.0\n<\/code><\/pre>\n<p>Note that the distance between two consecutive pixels here is always 0.5 (because <code>input_width \/ output_width = 10\/20 = 0.5<\/code>). This is true regardless of how big or small the image is: in other words, the step size is simply <code>1 \/ scale_factor<\/code>.<\/p>\n<p>With <code>align_corners=True<\/code>, however, the step size is slightly smaller than 0.5. It is in fact <code>(input_width - 1) \/ (output_width - 1)<\/code> or <code>9\/19 = 0.47368421<\/code> in the first example, and <code>19\/39 = 0.48717949<\/code> in the second.<\/p>\n<p>This explains why <code>align_corners=True<\/code> has a slightly different step size if the input size is different.<\/p>\n<p>Given the above, you might think that <code>align_corners=False<\/code> would actually give better, more consistent results, when I\u2019ve been telling you it doesn\u2019t?!<\/p>\n<p>But notice how pixel 9.0 actually appears twice at the end? That explains why the last column (and row) are duplicated. It also means all the pixels end up shifted to the top and left by a small amount. Very weird, and definitely not how this is supposed to work.<\/p>\n<p>Having a consistent step size is good, but <code>align_corners=False<\/code> gets the math wrong. And while <code>align_corners=True<\/code> looks better, its step size depends on the input dimensions. Neither is desirable.<\/p>\n<h3>align_corners=True in Core ML<\/h3>\n<p>You\u2019ll be happy to know that Core ML also supports this type of bilinear resizing.<\/p>\n<p>You cannot use the <code>Upsample<\/code> layer for this, you must use a <code>ResizeBilinear<\/code> layer in <code>STRICT_ALIGN_ENDPOINTS_MODE<\/code>. That will give exactly the same results as TensorFlow\u2019s bilinear resizing with <code>align_corners=True<\/code>.<\/p>\n<h3>align_corners=True in PyTorch<\/h3>\n<p>If you\u2019re using PyTorch, the story is also rather interesting. PyTorch has different APIs for upsampling. The preferred one is <code>torch.nn.functional.interpolate()<\/code>. All the other resizing APIs are actually implemented in terms of this <code>interpolate()<\/code> function (and are largely deprecated).<\/p>\n<p>Here\u2019s the confusing bit: PyTorch\u2019s <code>interpolate()<\/code> also has an <code>align_corners<\/code> property but it <em>only<\/em> works the same way as in TensorFlow if <code>align_corners=True<\/code>! The behavior for <code>align_corners=False<\/code> is completely different between PyTorch and TF.<\/p>\n<p>Up to version 0.3.1 of PyTorch, <code>align_corners=True<\/code> was the default. Since then, the default was switched to <code>align_corners=False<\/code>. If you have old PyTorch code, it may be assuming that this argument should be <code>True<\/code>. Beware!<\/p>\n<p>Some APIs such as <code>F.upsample_bilinear()<\/code> and the <code>nn.UpsamplingBilinear2d()<\/code> layer still set <code>align_corners<\/code> to True while other APIs set it to False. So you have to pay attention to which API you\u2019re using!<\/p>\n<p>Long story short: If your PyTorch model uses <code>align_corners=True<\/code>, you get the same results as TensorFlow. You can use a <code>ResizeBilinear<\/code> layer in your Core ML model in <code>STRICT_ALIGN_ENDPOINTS_MODE<\/code>.<\/p>\n<p>Plot twist: \u2026but if you do, it is actually <strong>impossible to convert your PyTorch model<\/strong> to Core ML using the existing tools. More about this later.<\/p>\n<p>What about using <code>align_corners=False<\/code> in PyTorch? I said it doesn\u2019t do the same (bad) thing as TensorFlow. Does that mean\u2026? Indeed, it does the correct thing, as we\u2019ll talk about next.<\/p>\n<h2>The good method<\/h2>\n<p>This is what the \u201ccorrect\u201d method produces:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/Bilinear@2x.jpg\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/Bilinear.jpg\" alt=\"The correct method of bilinear upsampling\"\/><\/a><br \/>\n<\/figure>\n<p>Not surprisingly, that\u2019s the same image we got from the Python code I showed earlier.<\/p>\n<p>If you compare this to the \u201cbad\u201d and the \u201cso-so\u201d methods, it looks much more stable \u2014 the pixels remain in their original place:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/NN+Correct.gif\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-upsampling\/NN+Correct.gif\" alt=\"Animation between nearest neighbors and the correct method\"\/><\/a><br \/>\n<\/figure>\n<div class=\"note\">\n<p><strong>Note:<\/strong> To be fair, we do lose the highlight from the yellow pixel in the top-left corner. That\u2019s because the center of that pixel is no longer the center of any of the upsampled pixels, and so the yellow color gets blended with the colors of its neighbors, which are all darker. Overall the image retains the same brightness but the highlights are distributed over multiple pixels now.<\/p>\n<\/div>\n<p>For the same input tensor we looked at before, upsampling by a factor of 2, we will read from the following x-coordinates:<\/p>\n<pre><code class=\"language-nohighlight\">-0.25, 0.25, 0.75, 1.25, 1.75, 2.25, 2.75, 3.25, 3.75, 4.25,  \n 4.75, 5.25, 5.75, 6.25, 6.75, 7.25, 7.75, 8.25, 8.75, 9.25\n<\/code><\/pre>\n<p>Just like with TensorFlow\u2019s <code>align_corners=False<\/code>, the distance between consecutive pixels is always 0.5. That\u2019s great, we wanted a consistent step size.<\/p>\n<p>However, we don\u2019t start and stop exactly on the corner points. Note that the first point, -0.25, is negative and the last point, 9.25, lies outside the feature map too. That\u2019s no problem, we can simply use the edge values for pixels that lie outside the feature map.<\/p>\n<p>Recall that <code>align_corners=True<\/code> had the issue that which points get sampled depends on the size of the input tensor. The correct method doesn\u2019t have this problem. It will always sample the same points, regardless of the input size.<\/p>\n<p>For example, let\u2019s say we have an image of width 20 and we want to upsample this by 2\u00d7. Now the following x-coordinates will be used:<\/p>\n<pre><code class=\"language-nohighlight\">-0.25,  0.25,  0.75,  1.25,  1.75,  2.25,  2.75,  3.25,  3.75,  4.25,  \n 4.75,  5.25,  5.75,  6.25,  6.75,  7.25,  7.75,  8.25,  8.75,  9.25,\n...\n14.75, 15.25, 15.75, 16.25, 16.75, 17.25, 17.75, 18.25, 18.75, 19.25\n<\/code><\/pre>\n<p>Notice how these pixels are spaced in the same way as before, there are just more of them. With this method, which points we sample depends only on the scaling factor, not on the sizes of the tensors.<\/p>\n<p>In <strong>PyTorch<\/strong>, you can do this kind of bilinear scaling with <code>F.interpolate(..., align_corners=False)<\/code>. As of PyTorch 0.3.1, this is the default behavior.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> But keep in mind that certain APIs such as <code>nn.UpsamplingBilinear2d()<\/code> still do <code>align_corners=True<\/code> under the hood.<\/p>\n<\/div>\n<p>In <strong>TensorFlow 2.x<\/strong>, you can use <code>tf.image.resize()<\/code> for this. If you use <code>tf.keras<\/code> in TF 2.x, the <code>UpSampling2D<\/code> layer also gets the correct behavior.<\/p>\n<p>This also is how other image processing libraries, such as OpenCV, do their bilinear scaling. So we\u2019re all finally on the same page\u2026<\/p>\n<p><strong>\u2026except for Core ML.<\/strong> Unfortunately, Core ML cannot do this currently.<\/p>\n<p>You read that right, Core ML does not have a bilinear resizing layer that does it \u201cthe correct way\u201d. This means Core ML\u2019s resizing layers are incompatible with PyTorch\u2019s <code>align_corners=False<\/code> and with TF 2.x.<\/p>\n<p>So for now, if you want your Core ML model to give 100% the same results as your PyTorch model, you have to use <code>align_corners=True<\/code>. In TensorFlow 2.x, you can use the old methods from <code>tf.compat.v1<\/code>, with <code>align_corners=True<\/code> or <code>False<\/code>.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> As I said in the introduction, it\u2019s usually OK if the bilinear resizing layer in your Core ML model is off by a few pixels. You\u2019ll most likely still get reasonable predictions out of your model. Just be aware that this difference exists and where it comes from.<\/p>\n<\/div>\n<h2>PyTorch conversion woes<\/h2>\n<p>Converting a PyTorch model to Core ML is usually done by exporting to ONNX first, then using the <strong>onnx_coreml<\/strong> package to do the conversion.<\/p>\n<p>Let\u2019s say you\u2019ve decided to use <code>align_corners=True<\/code> for your upsampling layers. (Note: I\u2019m not saying this is the correct choice. But right now it is a better choice than PyTorch\u2019s <code>align_corners=False<\/code> if you want the Core ML model to give 100% the same results as PyTorch.)<\/p>\n<p>Now you want to call <code>torch.export.onnx()<\/code> to convert the PyTorch model to ONNX format. Whoops, you\u2019ll probably get this warning message:<\/p>\n<pre><code class=\"language-nohighlight\">UserWarning: You are trying to export the model with onnx:Upsample\nfor ONNX opset version 9. This operator might cause results to not \nmatch the expected results by PyTorch. ONNX's Upsample\/Resize operator\ndid not match Pytorch's Interpolation until opset 11. Attributes to\ndetermine how to transform the input were added in onnx:Resize in opset\n11 to support Pytorch's behavior (like coordinate_transformation_mode \nand nearest_mode). We recommend using opset 11 and above for models \nusing this operator. \n<\/code><\/pre>\n<p>What\u2019s going on here?<\/p>\n<p>Like Core ML, ONNX has different versions. The version number determines which operators, or ops, are available. By default, when you export on ONNX graph from PyTorch it uses \u201copset\u201d version 9. However, this version has certain limitations, and one of those is that upsampling with <code>align_corners=True<\/code> is not supported.<\/p>\n<p>There is a newer version of the ONNX spec, opset 11, that is more capable. You can call <code>torch.export.onnx(..., opset_version=11)<\/code> to enable these new features \u2014 but unfortunately, this isn\u2019t very useful for us.<\/p>\n<h3>Opset v9<\/h3>\n<p>The ONNX <a href=\"https:\/\/github.com\/onnx\/onnx\/blob\/master\/docs\/Changelog.md#Upsample-9\">Upsample operator<\/a> in opset version 9 is fairly limited. It has a <code>mode<\/code> property for choosing between nearest and bilinear. It takes as input the source tensor and a <code>scales<\/code> tensor with the scaling factors for each dimension.<\/p>\n<p>You cannot export PyTorch models to ONNX using opset version 9 if they contain upsampling layers with <code>align_corners=True<\/code>. This gives an error message. You can only export if your upsampling layers use <code>align_corners=False<\/code> everywhere.<\/p>\n<p>Another thing to keep in mind is how you tell PyTorch\u2019s <code>interpolate()<\/code> function what the new tensor size is. You can either supply a scale factor or a width and height:<\/p>\n<p><strong>Using a scale factor.<\/strong> <code>F.interpolate(..., scale_factor=...)<\/code> will add a lot of extra operations to the ONNX graph, such as Gather, Cast, etc. These multiply the width and height of the input tensor by the scaling factors, floor the results, then divide by width and height again. The results are used as the scaling factors passed into the Upsample operation. (Not sure why they go through all this trouble.)<\/p>\n<p>Having those extra ops in the graph is not a problem for <strong>onnx_coreml<\/strong>. It will get rid of all those and just add a regular <code>UpsampleLayer<\/code> in the Core ML model.<\/p>\n<p><strong>Using a fixed output size.<\/strong> <code>F.interpolate(..., size=...)<\/code> does not add any extra ops. However, it doesn\u2019t actually pass the size to the Upsample operation in the ONNX graph \u2014 it has converted the size to a fixed scaling factor. Very convenient as that\u2019s exactly what Core ML needs, but pay attention:<\/p>\n<p>If the output size divided by the input size is not an integer, your Core ML model will now have an <code>Upsample<\/code> layer that has the wrong scaling factor. <strong>onnx_coreml<\/strong> will not warn about this happening! (Tested with onnx_coreml 1.2.)<\/p>\n<h3>Opset v11<\/h3>\n<p>In opset 11, there is a <a href=\"https:\/\/github.com\/onnx\/onnx\/blob\/master\/docs\/Operators.md\">Resize operator<\/a> that is much more powerful. It has:<\/p>\n<ul>\n<li>\n<p><code>coordinate_transformation_mode<\/code>. This supports <code>align_corners<\/code> mode, as well as several other modes, such as:<\/p>\n<ul>\n<li>for nearest neighbors: <strong>asymmetric<\/strong><\/li>\n<li>for bilinear, <code>align_corners=False<\/code>: <strong>pytorch_half_pixel<\/strong><\/li>\n<li>for bilinear, <code>align_corners=True<\/code>: <strong>align_corners<\/strong><\/li>\n<\/ul>\n<\/li>\n<li>\n<p><code>scales<\/code> or <code>sizes<\/code> tensor, so now <code>F.interpolate(..., size=...)<\/code> will actually keep the output size and not convert it to a scaling factor.<\/p>\n<\/li>\n<\/ul>\n<p>Problem: <strong>onnx_coreml<\/strong> currently does not support the Resize operator from opset v11 (tested with version 1.2). Trying to convert such a model to Core ML fails with the error:<\/p>\n<pre><code>NotImplementedError: Unsupported ONNX ops of type: Resize\n<\/code><\/pre>\n<p>Core ML 3 does have <a href=\"https:\/\/machinethink.net\/blog\/new-in-coreml3\/\">a lot of new operators<\/a> and you can enable these by running <code>onnx_coreml.convert(..., minimum_ios_deployment_target=\"13\")<\/code> but for me that still failed (could be a bug in the converter).<\/p>\n<h3>The workaround<\/h3>\n<p>There is no problem converting a PyTorch model to ONNX and then to Core ML if it uses <code>align_corners=False<\/code>. But as we discussed, if you train your model with <code>align_corners=False<\/code> \u2014 even if that is the correct way to do bilinear resizing \u2014 then Core ML won\u2019t work 100% the same way.<\/p>\n<p>And if you train with <code>align_corners=True<\/code>, Core ML will work correctly\u2026 but you can\u2019t convert the PyTorch model! Catch-22.<\/p>\n<p>The workaround is to train with <code>align_corners=True<\/code>, then edit your Python code to temporarily turn this into <code>False<\/code> in order to do the export.<\/p>\n<p>Next, export the model to ONNX using <code>torch.export.onnx(model, ..., opset_version=9)<\/code>.<\/p>\n<p>Then, use <strong>onnx_coreml<\/strong> to convert the ONNX model to Core ML. This will give you a Core ML model with an <code>Upsample<\/code> layer (possibly with incorrect scaling factors).<\/p>\n<p>Finally, use coremltools to manually replace this by a <code>ResizeBilinear<\/code> layer with mode <code>STRICT_ALIGN_ENDPOINTS_MODE<\/code>. Now your Core ML model will give the exact same results as your PyTorch model.<\/p>\n<div class=\"note\">\n<p><strong>Tip:<\/strong> You can learn how to do this kind of model surgery in my <a href=\"https:\/\/leanpub.com\/coreml-survival-guide\/\">Core ML Survival Guide<\/a> e-book.<\/p>\n<\/div>\n<h3>No dynamic scale factors<\/h3>\n<p>Something else to keep in mind: PyTorch is very dynamic, meaning that the scaling factor and\/or target size can be computed at runtime from the properties of some tensor.<\/p>\n<p>For example, if tensor <code>a<\/code> is the output of a conv layer and you want to resize another tensor <code>b<\/code> to the width and height of <code>a<\/code>. Depending on the shape of the input to the conv layer, the size of <code>a<\/code> can be different each time. Easy to do in PyTorch, impossible in Core ML.<\/p>\n<p>In Core ML, the scaling factor and target size are always a hardcoded number inside the mlmodel file. The conversion to ONNX or Core ML may not be able to handle scaling factors or target sizes that are computed dynamically.<\/p>\n<h2>Cheatsheet<\/h2>\n<p>All right, no one can remember all that, so here\u2019s a cheatsheet of what works and what doesn\u2019t.<\/p>\n<h4>The bad but backwards compatible way<\/h4>\n<p>If you have,<\/p>\n<ul>\n<li>TensorFlow 1.x model with <code>align_corners=False<\/code><\/li>\n<li>standalone Keras with TF backend, <code>UpSampling2D<\/code> layer<\/li>\n<\/ul>\n<p>then use:<\/p>\n<ul>\n<li><code>Upsample<\/code> layer in mode <code>\"BILINEAR\"<\/code><\/li>\n<li><code>ResizeBilinear<\/code> layer with <code>UPSAMPLE_MODE<\/code><\/li>\n<\/ul>\n<h4>The slightly better way<\/h4>\n<p>If you have,<\/p>\n<ul>\n<li>TensorFlow 1.x model with <code>align_corners=True<\/code><\/li>\n<li>PyTorch model with <code>align_corners=True<\/code><\/li>\n<\/ul>\n<p>then use:<\/p>\n<ul>\n<li><code>ResizeBilinear<\/code> layer in <code>STRICT_ALIGN_ENDPOINTS_MODE<\/code><\/li>\n<\/ul>\n<h4>The best way<\/h4>\n<p>If you have,<\/p>\n<ul>\n<li>TensorFlow 2.x model with <code>tf.image.resize()<\/code><\/li>\n<li><code>tf.keras.layers.UpSampling2D<\/code> layer<\/li>\n<li>PyTorch with <code>align_corners=False<\/code><\/li>\n<\/ul>\n<p>then there is no matching solution in Core ML. \ud83d\ude22<\/p>\n<p>In this case, I suggest you try them all \u2014 the <code>Upsample<\/code> layer, or a <code>ResizeBilinear<\/code> layer in <code>UPSAMPLE_MODE<\/code> or <code>STRICT_ALIGN_ENDPOINTS_MODE<\/code> \u2014 and see which one gives the best results on your data.<\/p>\n<p>Personally, I\u2019d probably avoid <code>UPSAMPLE_MODE<\/code> in this situation because it shifts the pixels. If your model has several upsampling layers in succession, for example in a segmentation model, the repeated shifts may cause the final predicted mask to be off by multiple pixels.<\/p>\n<div class=\"note-update\">\n<p><strong>Update 4 May 2020:<\/strong> An eagle-eyed reader pointed out that the plots of the upsampled images were incorrect. Unless you tell it otherwise, matplotlib automatically normalizes the pixel intensities, which made the upsampled images seem brighter than they should be. I fixed this by plotting all images using the same range of pixel intensities.<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Resizing feature maps is a common operation in many neural networks, especially those that perform some kind of image segmentation task. One issue I ran into recently while converting a neural network to Core ML, is that the original PyTorch model gave different results for its bilinear upsampling than Core ML, and I wanted to [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":10199,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rop_custom_images_group":[],"rop_custom_messages_group":[],"rop_publish_now":"initial","rop_publish_now_accounts":[],"rop_publish_now_history":[],"rop_publish_now_status":"pending","footnotes":""},"categories":[1],"tags":[],"class_list":["post-10198","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-explore"],"_links":{"self":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/posts\/10198","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/comments?post=10198"}],"version-history":[{"count":0,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/posts\/10198\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/media\/10199"}],"wp:attachment":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/media?parent=10198"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/categories?post=10198"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/tags?post=10198"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}