I see the following questions come up a lot on Stack Overflow, the Apple developer forums, and various Slack groups:<\/p>\n
\nMy neural network works on images but the Core ML model expects an MLMultiArray object. How do I convert my UIImage to an MLMultiArray?<\/p>\n
My neural network outputs an MLMultiArray but how do I convert this back into a UIImage?<\/p>\n
My model outputs an image but the UIImage is all black.<\/p>\n<\/blockquote>\n
People run into these issues because most training tools, such as Keras or PyTorch, treat images just like any other data \u2014 as an n-dimensional array.<\/p>\n
But in Core ML images are special!<\/strong><\/p>\n
A common issue is that, after converting a model to Core ML, the data type of the input is \u201cmulti-array\u201d and not \u201cimage\u201d. This is often easily fixed, but sometimes it requires a bit of model surgery.<\/p>\n
How to properly handle images with your Core ML models is described in detail in my e-book Core ML Survival Guide<\/a> \u2014 along with many other handy tips & tricks! Because the topic comes up so often, I decided to write about it on my blog as well.<\/p>\n
If you\u2019re having problems using images in Core ML, read on!<\/p>\n
Image as input<\/h2>\n
The situation: You have a Core ML model with an input<\/strong> that is a multi-array of shape (3, height, width)<\/em>.<\/p>\n
But in your app you have an image object, such as a
UIImage<\/code>.<\/p>\nHow do you convert the
UIImage<\/code> to anMLMultiArray<\/code> object that you can pass to the model?<\/p>\nThe answer is: You probably shouldn\u2019t do this!<\/p>\n
A better solution is to change the model to expect an image<\/strong> instead of a multi-array. Now Core ML will directly accept your
UIImage<\/code> as the input \u2014 no need to convert it to anMLMultiArray<\/code> first.<\/p>\nIf a model works on images, it should accept images as input! Multi-arrays are useful for other kinds of data, but are not intended for images.<\/p>\n
You can probably do this during conversion<\/h3>\n
This problem is easy to fix if you have access to the original model.<\/p>\n
Most Core ML converters have an
image_input_names<\/code> option that tells the model which inputs should be treated as an image.<\/p>\nIn your conversion script, simply provide the additional argument
image_input_names=\"your_input\"<\/code> to theconvert<\/code> function, where\"your_input\"<\/code> is the name of the input that should take an image.<\/p>\nFor a model with multiple image inputs, you can supply a list of names:
image_input_names=[\"first_input\", \"second_input\"]<\/code>.<\/p>\nNote that this only works if the shape of the input is (3, height, width)<\/em> for color images, or just (height, width)<\/em> for grayscale images.<\/p>\n
\nNote:<\/strong> Core ML doesn\u2019t currently work with 4 channel images. The alpha-channel in such images is simply ignored.<\/p>\n<\/div>\n
Changing the mlmodel file afterwards<\/h3>\n
If you only have the .mlmodel<\/strong> file and not the original model or the conversion script \u2014 or if the converter does not have the
image_input_names<\/code> option \u2014 you can still fix this using a bit of Python.<\/p>\nimport coremltools\nimport coremltools.proto.FeatureTypes_pb2 as ft \n\nspec = coremltools.utils.load_spec(\"YourModel.mlmodel\")\n\ninput = spec.description.input[0]\ninput.type.imageType.colorSpace = ft.ImageFeatureType.RGB\ninput.type.imageType.height = 224 \ninput.type.imageType.width = 224\n\ncoremltools.utils.save_spec(spec, \"YourNewModel.mlmodel\")\n<\/code><\/pre>\nThis script changes the data type of the first input,
input[0]<\/code>, to expect an image.<\/p>\nThis assumes that the image is RGB. If necessary, change
colorSpace<\/code> toGRAYSCALE<\/code> orBGR<\/code>. You should use BGR if the model was trained with Caffe or if OpenCV was used to load the training images. Core ML will automatically convert the pixel order of your input images to whatever format the model requires, but it can only do this if you setcolorSpace<\/code> correctly.<\/p>\nAlso make sure that the width and height are correct for your model!<\/p>\n
\nTip:<\/strong> To support flexible image sizes, see the chapter Size Flexibility<\/em> in the Core ML Survival Guide<\/a>.<\/p>\n<\/div>\n
Don\u2019t forget preprocessing<\/h3>\n
A Core ML model includes a special preprocessing stage<\/strong> for image inputs.<\/p>\n
Usually the original model normalizes the image tensor. For example, by converting the pixels from the range [0, 255] to [-1, +1].<\/p>\n
You need to set this up in your Core ML model as well. Getting the preprocessing wrong is the number one reason why people get incorrect predictions out of their Core ML models!<\/p>\n
To learn more about this, see my blog post Help!? The output of my Core ML model is wrong\u2026<\/a> And of course you can read all about this in the Core ML Survival Guide<\/a>.<\/p>\n
You need a CVPixelBuffer<\/h3>\n
OK, I lied a little when I said that Core ML could directly use
UIImage<\/code> objects. \ud83d\ude30<\/p>\nThe Core ML API requires images to be
CVPixelBuffer<\/code> objects. But in the app you probably have the image as aUIImage<\/code>, aCGImage<\/code>, aCIImage<\/code>, or anMTLTexture<\/code>.<\/p>\nIn that case, you will still need to convert your image to a
CVPixelBuffer<\/code> object. Fortunately, this is easy:<\/p>\n\n
- \n
On iOS 13 and macOS 10.15 you can use the new
MLFeatureValue(cgImage:)<\/code> andMLFeatureValue(imageAt:)<\/code> APIs.<\/p>\n<\/li>\n- \n
You can use
VNCoreMLRequest<\/code> from the Vision framework, which lets you pass in aCGImage<\/code> object that you can easily obtain from yourUIImage<\/code>.<\/p>\n<\/li>\n- \n
Or you can use the helper code from CoreMLHelpers<\/a>.<\/p>\n<\/li>\n<\/ul>\n
Converting UIImage \u2192 MLMultiArray<\/h3>\n
If \u2014 for some strange reason \u2014 you really must<\/em> convert the
UIImage<\/code> into anMLMultiArray<\/code> anyway, here\u2019s how to do it:<\/p>\n\n
- \n
Create an
MLMultiArray<\/code> of type.double<\/code> with shape (3, height, width)<\/em>.<\/p>\n<\/li>\n- \n
Loop through the pixels in the image and copy the color values into the rows and columns of this new
MLMultiArray<\/code> object.<\/p>\n<\/li>\n- \n
Don\u2019t forget the preprocessing! You can do this inside the loop. For example, to convert a pixel from [0, 255] to [-1, 1], first divide the pixel value by 127.5, then subtract 1, and put the resulting value into the
MLMultiArray<\/code>.<\/p>\n<\/li>\n<\/ul>\nI don\u2019t have any code for this (because it\u2019s a bad idea) but it\u2019s basically the opposite of what happens in MLMultiArray+Image.swift in CoreMLHelpers<\/a>.<\/p>\n
More trouble than it\u2019s worth, really \u2014 it\u2019s much smarter to let Core ML handle this for you. \ud83d\ude01<\/p>\n
Images as output<\/h2>\n
What if your model outputs<\/strong> an image?<\/p>\n
As described above, the converters provided with coremltools have an option
image_input_names<\/code> that tells the converter which of the inputs<\/em> should be treated as images, so that Core ML lets you pass in aCVPixelBuffer<\/code> object.<\/p>\nHowever, there is no
image_output_names<\/code>. So if you have an image-to-image model, any image outputs will become multi-array outputs<\/strong> in the mlmodel. That\u2019s not handy.<\/p>\nIf a model predicts an image, it should output some kind of image object too! In Core ML, that would be a
CVPixelBuffer<\/code> object.<\/p>\nThere are two things you can do to get a
CVPixelBuffer<\/code> as output from Core ML:<\/p>\n\n
- convert the
MLMultiArray<\/code> to an image yourself, or<\/li>\n- change the mlmodel so that it knows the output should be an image.<\/li>\n<\/ol>\n
I recommend against using option 1. It is slow and unnecessary because you can let Core ML do it for you (option 2). But in case you want to do the conversion yourself, check out the MLMultiArray+Image extension in CoreMLHelpers<\/a>.<\/p>\n
Changing the output type to image<\/h3>\n
Even though the converter doesn\u2019t let you specify that an output should be an image, you can always change this in the mlmodel file afterwards.<\/p>\n
Load the mlmodel into a spec object:<\/p>\n
import coremltools\nimport coremltools.proto.FeatureTypes_pb2 as ft \n\nspec = coremltools.utils.load_spec(\"YourModel.mlmodel\")\n<\/code><\/pre>\nLet\u2019s say that
print(spec.description)<\/code> gives the following:<\/p>\noutput {\n name: \"generatedImage\"\n type {\n multiArrayType {\n shape: 3\n shape: 300\n shape: 150\n dataType: DOUBLE\n }\n }\n}\n<\/code><\/pre>\nThen you can turn this output description into an image by writing:<\/p>\n
output = spec.description.output[0]\n\nimport coremltools.proto.FeatureTypes_pb2 as ft\noutput.type.imageType.colorSpace = ft.ImageFeatureType.RGB\noutput.type.imageType.height = 300\noutput.type.imageType.width = 150\n\ncoremltools.utils.save_spec(spec, \"YourNewModel.mlmodel\")\n<\/code><\/pre>\nAnd now
print(spec.description)<\/code> shows the output is an RGB image:<\/p>\noutput {\n name: \"generatedImage\"\n type {\n imageType {\n width: 150\n height: 300\n colorSpace: RGB\n }\n }\n}\n<\/code><\/pre>\nIf you want the image to have BGR pixel order, write
colorSpace = ft.ImageFeatureType.BGR<\/code> instead ofRGB<\/code>.<\/p>\nImportant:<\/strong> You can\u2019t just turn any<\/em> multi-array into an image. Pay attention to the
shape<\/code> properties in themultiArrayType<\/code> structure:<\/p>\n\n
- the first
shape<\/code> is the number of channels<\/li>\n- the second is the image\u2019s height<\/li>\n
- the third is the image\u2019s width<\/li>\n<\/ul>\n
You can only turn a multi-array into an RGB or BGR image if the number of channels is 3.<\/p>\n
If the number of channels is 1, or if there are only two
shape<\/code> values \u2014 height and width \u2014 then you must usecolorSpace = ft.ImageFeatureType.GRAYSCALE<\/code>.<\/p>\nIf there are more than three
shape<\/code> values listed, and the first or last of these are 1, then you can delete these unused dimensions. For example, if you have:<\/p>\nmultiArrayType {\n shape: 1\n shape: 3\n shape: 300\n shape: 150\n dataType: DOUBLE\n}\n<\/code><\/pre>\nThen do the following to get rid of that first dimension with size 1:<\/p>\n
del output.type.multiArrayType.shape[0]\n<\/code><\/pre>\nRemoving a dimension of size 1 doesn\u2019t change the actual data, and now it lets Core ML interpret the data as being an image.<\/p>\n
Don\u2019t forget the postprocessing<\/h3>\n
If your Core ML model includes preprocessing, for example to convert the pixels from the range [0, 255] to [-1, +1], it\u2019s likely that the output of your model is also in that [-1, +1] range.<\/p>\n
For Core ML to turn the output data into a
CVPixelBuffer<\/code>, it must be a tensor with values in the range 0 \u2013 255. Larger values or negative values will be clipped to this range.<\/p>\nThis means that, if your model doesn\u2019t already output pixels from 0 \u2013 255, you\u2019ll need to add some postprocessing layers<\/strong>.<\/p>\n
For outputs in the range [-1, +1], you\u2019ll need to add two operations:<\/p>\n
\n
- add 1 to the tensor to put it in the range [0, 2]<\/li>\n
- multiply the tensor by 127.5<\/li>\n<\/ol>\n
This is easiest if you add these operations in the original model and then convert it to Core ML.<\/p>\n
In case you don\u2019t have access to the original model, you can also change this directly in the mlmodel file by adding new layers at the end<\/strong>. For more info on how to insert new layers into an existing mlmodel file, see the Core ML Survival Guide<\/a>.<\/p>\n
If you forget to do this postprocessing, your image will be black or the colors will be wrong in some other weird way. Usually the postprocessing needs to be the inverse of the preprocessing.<\/p>\n
You get a CVPixelBuffer<\/h3>\n
For image outputs, Core ML gives you a
CVPixelBuffer<\/code> object.<\/p>\nIf you used Vision, you\u2019ll get a
VNPixelBufferObservation<\/code> object that contains aCVPixelBuffer<\/code>.<\/p>\nFortunately, a
CVPixelBuffer<\/code> is easy enough to convert into aUIImage<\/code>. Here is one way, using Core Image:<\/p>\nlet ciImage = CIImage(cvPixelBuffer: pixelBuffer)\nlet resultImage = UIImage(ciImage: ciImage)\n<\/code><\/pre>\nConclusion<\/h2>\n
Core ML can work with image inputs and outputs just fine, but remember:<\/p>\n
\n
- you need to set the type of the inputs and\/or outputs to accept images instead of multi-arrays<\/li>\n
- you need to add preprocessing to normalize the pixel values<\/li>\n
- for models with image outputs, you probably need to add postprocessing too<\/li>\n<\/ul>\n
Letting Core ML handle the conversion from
CVPixelBuffers<\/code> to tensors and back is a lot easier \u2014 and more efficient \u2014 than doing converting your images to\/fromMLMultiArray<\/code> objects by hand!<\/p>\n