![\"WWDC](\"https:\/\/machinethink.net\/images\/coreml-new-2020\/wwdc2020.png\")
<\/a>\n<\/figure>\n
In this blog post, I\u2019ll summarize what\u2019s new<\/strong> in Core ML and the other AI and ML technologies from the Apple ecosystem.<\/p>\n Last year was a big update for Core ML, but this year the improvements are much more modest: a few new layer types, support for encrypting models, and the ability to host model updates on CloudKit.<\/p>\n It looks like the version number got dropped. Last year\u2019s update was called Core ML 3, but now the name is simply Core ML without a number. However, coremltools did get bumped to version 4.<\/p>\n Note:<\/strong> The internal mlmodel specification version number is now 5, so new models will show up as \u201cCore ML v5\u201d in Netron<\/a>.<\/p>\n<\/div>\n The new layers are:<\/p>\n These layer types can only be used with specification version 5 or greater, which means iOS 14 and macOS 11.0 or higher.<\/p>\n Another useful improvement is 8-bit quantized operations<\/strong> for the following layers:<\/p>\n In previous versions of Core ML you could quantize the weights, but they would be dequantized back to floating point when the model got loaded. With the new Computations with INT8 have the potential to be much faster than floating point operations, but I wonder if that is only an advantage for the CPU and not the GPU, since GPUs really like floating point. Perhaps the upcoming Neural Engine update will support INT8 operations natively. (Apple did buy Xnor.ai<\/a> a while back\u2026)<\/p>\n Core ML on the CPU can now also use 16-bit floating point<\/strong> operations instead of 32-bit floats (on A11 Bionic and up). As mentioned in the Explore numerical computing in Swift<\/a> video, Note:<\/strong> Core ML already used Other (small) changes:<\/p>\n There appear to be no changes to on-device training: still only fully-connected and convolution layers are supported. The The following new model types<\/strong> were added:<\/p>\n This new Core ML feature lets you update your models independently of the app.<\/p>\n Instead of having to publish an app update, the existing app can simply download a new version of the mlmodel file. To be fair, this is not a new idea and several third-party vendors have SDKs for this. It\u2019s also not that hard to build this yourself. The advantage of using Apple\u2019s solution is that the models get hosted on Apple Cloud<\/strong>.<\/p>\n Because you may have more than one model in your app, the new concept of model collections<\/strong> lets you bundle multiple models together and the app will update all of them at once. You create these collections on the CloudKit dashboard.<\/p>\n In the app, you use the To prepare a Core ML model for deployment, there is now a Create Model Archive<\/strong> button in Xcode. This writes a .mlarchive<\/strong> file. You can upload this version of the model to your CloudKit dashboard and then put it into a model collection. (It looks like mlarchive is really just a zip file with the contents of your mlmodelc folder.)<\/p>\n A nice feature is that you can deploy different model collections to different users. For example, the camera on the iPhone is different than the camera on the iPad, so you may want to create two versions of a model and send one to iPhone users of the app and the other to iPad users.<\/p>\n You can create targeting rules for the device class (iPhone \/ iPad \/ TV \/ Watch), operating system and version, region code, language code, and app version.<\/p>\n There appears to be no mechanism for dividing your users into groups by other criteria, such as for A\/B testing model updates, or for targeting specific device types such as \u201ciPhone X or lower\u201d. However, you can still do this manually by creating collections with different names and then explicitly asking Deploying a new model version isn\u2019t always immediate<\/strong>. The app will at some point detect that a new model is available, and will automatically download it and place it in the app\u2019s sandbox. But you don\u2019t get any control over when or how this happens \u2014 Core ML might perform the download in the background while the phone is not being used, for example.<\/p>\n Because of this, it\u2019s a good idea to already ship your app with a built-in model as a fallback option, such as a generic model that handles both iPhone and iPad.<\/p>\n Even though this is a convenient solution and you don\u2019t have to worry about hosting the models yourself, do keep in mind that your app is now using CloudKit. From what I understood, the model collections do count towards your storage quotas and downloading the models counts towards your network traffic quotas.<\/p>\n See also:<\/p>\n Note:<\/strong> If you want to do on-device personalization of the model, there\u2019s no easy way to combine that with this new CloudKit update feature. There is no straightforward way to transfer what the personalized model has learned into the new model, or to combine these models somehow.<\/p>\n<\/div>\n Until now, it was really easy for someone to steal your Core ML model and stick it into their own app. As of iOS 14 \/ macOS 11.0, Core ML can automatically encrypt and decrypt models so that strangers can no longer peek inside your mlmodelc folders. You can use encryption with or without the new CloudKit deployment.<\/p>\n Xcode encrypts the compiled model, mlmodelc<\/strong>, not the original mlmodel file. The model always stays in encrypted form on the user\u2019s device. Only when the app instantiates the model, Core ML automatically decrypts it. This decrypted version only exists in memory, it is not stored as a file anywhere.<\/p>\n First, you need an encryption key. The good news is that you don\u2019t need to manage this key yourself! There is now a Create Encryption Key<\/strong> button in Xcode\u2019s Core ML model viewer. When you click this, Xcode generates a new encryption key that is associated with your Apple developer team account. No need to mess around with CSRs and Keychain Access for this one. \ud83e\udd2a<\/p>\n This process creates a new .mlmodelkey<\/strong> file. This key is stored on Apple\u2019s servers but you also get a local copy that you need for encrypting the models in Xcode. You don\u2019t need to embed this encryption key in your app \u2014 and shouldn\u2019t!<\/p>\n To encrypt a Core ML model, you can add To decrypt the model when the app instantiates it, Core ML will need to fetch the encryption key over the network<\/strong> from Apple\u2019s servers, so that obviously requires a network connection. Core ML only needs to do this the very first time you\u2019re using that model.<\/p>\n Naturally, if the network is down and the encryption key hasn\u2019t been downloaded yet, your app won\u2019t be able to instantiate the Core ML model. For this reason, you should use the new Pretty cool feature \u2014 if you\u2019re worried that people will steal your proprietary tech \u2014 and fairly easy to implement in your app.<\/p>\n See also:<\/p>\n Note:<\/strong> According to this forum post<\/a>, encrypted models cannot be used with on-device personalization. Makes sense.<\/p>\n<\/div>\n The iOS API for using Core ML models<\/a> hasn\u2019t changed much. Still, there are a few interesting things to point out.<\/p>\n The only new class is As you know, when you add an mlmodel file to your project, Xcode automatically generates a Swift or Objective-C source file with classes that make it easier to use the model. There are a couple of changes in these generated classes:<\/p>\n The default If your model works on images, instead of having to deal with A new warning has appeared in the MLModel docs<\/a>:<\/p>\n Use an MLModel instance on one thread or one dispatch queue at a time. You do this by either serializing method calls to the model, or by creating a separate model instance for each thread and dispatch queue.<\/p>\n<\/blockquote>\n Whoops, I\u2019m guilty of doing this. \ud83d\ude48 I was led to believe that Note:<\/strong> The Xcode now shows a lot more information about models, such as the class labels and any custom metadata that you added to the model. It also shows statistics about the types of layers inside the model.<\/p>\n There is a handy interactive preview<\/strong> that lets you play with the model without having to run the app. You can drag-and-drop images, videos, or text into this preview window and immediately view the model\u2019s predictions. Very nice!<\/p>\n You can also use Core ML models in playgrounds<\/strong> now. Xcode will auto-generate the class for it and you can use it as usual. That\u2019s another way you can interactively play with the model before putting it into the app.<\/p>\n Building your own models with Create ML is fun for simple projects, but it\u2019s much more common to train in TensorFlow or PyTorch. To use such a model with Core ML, you need to convert it to the mlmodel file format first. That\u2019s what coremltools is for.<\/p>\n Good news: the documentation is greatly improved<\/a>. You should check it out. Let\u2019s hope they keep this user guide up-to-date because the old documentation wasn\u2019t always the latest version.<\/p>\n Note:<\/strong> Unfortunately, the Jupyter notebook examples have disappeared. These examples are now part of the user\u2019s guide, but no longer as notebooks.<\/p>\n<\/div>\n The way to convert models has changed<\/strong> quite a bit. The old neural network converters are pretty much deprecated and have been replaced by a newer, more flexible design.<\/p>\n There are now three types of converters:<\/p>\n To convert a TensorFlow 1.x or 2.x, PyTorch, or tf.keras model, you use the new unified conversion API<\/strong>. It looks like this:<\/p>\n The This new conversion API converts the model into an intermediate representation<\/a> called MIL<\/strong>. There are currently converters for TensorFlow 1.x to MIL, TensorFlow 2.x to MIL \u2014 including tf.keras \u2014 and PyTorch to MIL. If at some point a new deep learning framework becomes popular, it can get its own MIL converter.<\/p>\n Once the model is in MIL format, it can be optimized according to general rules, such as stripping out unnecessary operations or fusing different layers together. Finally, it is converted from MIL to the mlmodel format.<\/p>\n I haven\u2019t looked at it in detail yet, but this new approach makes me believe that coremltools 4 produces much more efficient mlmodel files than before, especially for TF 2.x graphs.<\/p>\n What I like about MIL is that it allows you to tell the converter how to deal with layers that it doesn\u2019t understand yet<\/strong>. If your model has a layer that is not directly supported by Core ML, you might be able to split it up into more primitive MIL operations, such as matrix multiplications or other arithmetic.<\/p>\n The converter can then use this so-called \u201ccomposite operation\u201d whenever it encounters a layer of that type. That\u2019s way easier than implementing unsupported operations with a custom layer, although that\u2019s still possible. The documentation has a nice example<\/a> of how to use such composite operations.<\/p>\n See also:<\/p>\n There are several high-level frameworks in the iOS and macOS SDKs that perform machine learning-related tasks. Let\u2019s quicky look at what\u2019s new here.<\/p>\n The Vision framework<\/a> for computer vision got a bunch of new functionality.<\/p>\n Vision already had detection models for faces, face landmarks and human bodies. This new release adds the following:<\/p>\n It\u2019s actually really cool of Apple to include pose estimation in the OS. There are several open source models for this but they aren\u2019t really that great, or are slow, while commercial solutions are expensive. Now you get high-quality body pose detection for free!<\/p>\n Rather than just looking at static images, there is now also more emphasis on detecting things in video<\/strong>, both offline and from the camera in real-time. For convenience, you can use New is Once the thing you\u2019re looking for has been observed, the completion handler is called with a You don\u2019t use To perform offline analysis of videos, you can use An important traditional computer vision technique for analyzing video is optical flow<\/strong>. Vision now has Also new is The final new thing in Vision is a new variant of the built-in feature extractor, VisionFeaturePrint. iOS already came with VisionFeaturePrint.Scene, which is useful for making image classifiers. There is now a new VisionFeaturePrint.Object<\/strong> model that is optimized for extracting features that are useful for object detection.<\/p>\n It works on a 299\u00d7299 input image and outputs two multi-arrays of shape (288, 35, 35) and (768, 17, 17), respectively. These are not bounding box predictions yet, only \u201craw\u201d features. A full object detector still needs to add logic to turn these features into bounding boxes and class labels. Create ML does that when you train an object detector using transfer learning.<\/p>\n See also:<\/p>\n For NLP tasks, you can use the Natural Language framework. It works closely together with models trained in Create ML.<\/p>\n Only a few new features this year:<\/p>\nCore ML<\/h2>\n
New layer types in Core ML<\/h3>\n
\n
Convolution3DLayer<\/code>, Pooling3DLayer<\/code>, GlobalPooling3DLayer<\/code>: these are especially useful for processing video data, something you can now do with the Vision framework. (Core ML still has no explicit support for 1D convolutions, although you can use the regular 2D convolution layer for that.)<\/p>\n<\/li>\nOneHotLayer<\/code>: for one-hot encoding the input.<\/p>\n<\/li>\nClampedReLULayer<\/code>: a ReLU activation that has a maximum value (useful for making ReLU6).<\/p>\n<\/li>\nArgSortLayer<\/code>: for sorting the input tensor. This returns the sorted indices instead of the actual sorted values. There is no regular sorting layer, but you could use a GatherLayer<\/code> to reorder the elements from the argsort output.<\/p>\n<\/li>\nCumSumLayer<\/code>: computes the cumulative sum of the input tensor.<\/p>\n<\/li>\nSliceBySizeLayer<\/code>: Core ML already had several types of slicing layers. This one lets you pass in a tensor containing the index to start slicing from; the size of the slice is always fixed.<\/p>\n<\/li>\n<\/ul>\n\n
InnerProductLayer<\/code><\/li>\nBatchedMatMulLayer<\/code><\/li>\n<\/ul>\nint8DynamicQuantize<\/code> feature, the weights stay 8-bit integer values and the actual computations are performed using integers as well.<\/p>\nFloat16<\/code> is now a first-class Swift data type. With 16-bit floats supported natively on the CPU, Core ML can be more than twice as fast!<\/p>\nFloat16<\/code> on the GPU and Neural Engine, so this only makes a difference when using the CPU.<\/p>\n<\/div>\n\n
UpsampleLayer<\/code> can now have a fractional scaling factor. In BILINEAR<\/code> mode, there are new options for how the grid points are sampled (\u201calign corners\u201d). This should solve most of the issues I pointed out in this blog post<\/a>.<\/p>\n<\/li>\nReorganizeDataLayerParams<\/code> has a PIXEL_SHUFFLE<\/code> mode. This is another method for doing upsampling. Previously you could implement pixel shuffle using several permute and reshape layers but it\u2019s cool to have it built-in now.<\/p>\n<\/li>\nSliceStaticLayer<\/code> and SliceDynamicLayer<\/code> now have a squeezeMasks<\/code> property for even more slicing fun.<\/p>\n<\/li>\nTileLayer<\/code> accepts a second input tensor, so that you can specify the number of repetitions dynamically.<\/p>\n<\/li>\n<\/ul>\nMLParameterKey<\/code> class in CoreML.framework now has a configuration option for the RMSprop<\/strong> optimizer but this is currently not listed in NeuralNetwork.proto<\/a>. It might be added in a later beta.<\/p>\n\n
VisionFeaturePrint.Object<\/code>, a feature extractor that is optimized for doing object detection.<\/p>\n<\/li>\nSerializedModel<\/code>. Not sure what this is for. This is a \u201cprivate\u201d definition that is \u201csubject to change with no notice or support\u201d. Maybe this is a way for Apple to embed proprietary model formats inside an mlmodel?<\/p>\n<\/li>\n<\/ul>\nHosting model updates on CloudKit<\/h3>\n
<\/a>
\n<\/figure>\nMLModelCollection<\/code> class to download and manage the model updates. The WWDC video<\/a> shows code snippets for this.<\/p>\nMLModelCollection<\/code> to fetch the appropriate collection by name at runtime.<\/p>\n\n
Model encryption<\/h3>\n
<\/a>
\n<\/figure>\n--encrypt YourModel.mlmodelkey<\/code> to the compiler flags for the model, or, if you\u2019re going to use CloudKit deployment, you need to supply the encryption key when creating the model archive.<\/p>\nYourModel.load()<\/code> function. This has a completion handler that lets you respond to loading errors. For example, the modelKeyFetch<\/code> error code tells you that Core ML was unable to load the decryption key from Apple\u2019s servers.<\/p>\n\n
CoreML.framework<\/h3>\n
MLModelCollection<\/code>, which is for CloudKit deployment.<\/p>\n\n
init()<\/code> is now deprecated, so writing let model = YourModel()<\/code> will result in a compiler warning. Instead, use YourModel(configuration:)<\/code> or the new YourModel.load()<\/code> method, which lets you handle model loading errors such as when an encrypted model cannot be decrypted.<\/p>\n<\/li>\nCVPixelBuffer<\/code> objects, you can now create a YourModelInput<\/code> object using a CGImage<\/code> or the URL to a local PNG or JPG file, and do the prediction on that. However, this doesn\u2019t let you set the cropAndScale<\/code> method or the cropRect<\/code>, so if you want more control over how the images are resized, these new methods aren\u2019t very useful.<\/p>\n<\/li>\n<\/ul>\n\n
MLModel<\/code> internally used a serial queue for handling requests \u2014 but this may not have been true, or perhaps it has changed. Anyway, best to follow the advice from now on.<\/p>\nMLMultiArray<\/code> has a new init(concatenating:axis:dataType:)<\/code> initializer, which makes a new multi-array by concatenating several existing multi-arrays. These must all have the same shape except for the specified axis, along which they\u2019ll be concatenated. It looks like this was added specifically for doing predictions on videos, as with the new Action Classifier models from Create ML. Handy!<\/p>\nMLMultiArrayDataType<\/code> enum now has static .float<\/code> and .float64<\/code> properties. Not sure what these are for, as it already had .float32<\/code> and .double<\/code> cases. Beta glitch?<\/p>\n<\/div>\nXcode model viewer<\/h3>\n
<\/a>
\n<\/figure>\ncoremltools 4<\/h3>\n
\n
import coremltools as ct\n\nclass_labels = [ \"cat\", \"dog\" ]\n\nimage_input = ct.ImageType(shape=(1, 224, 224, 3),\n bias=[-1, -1, -1],\n scale=2\/255.)\n\nmodel = ct.convert(\n keras_model, \n inputs=[ image_input ],\n classifier_config=ct.ClassifierConfig(class_labels)\n)\n\nmodel.save(\"YourModel.mlmodel\")\n<\/code><\/pre>\nct.convert()<\/code> function inspects the model file to see what format it is and automatically chooses the appropriate converter. The arguments are somewhat different than before: the preprocessing arguments are passed in using an ImageType<\/code> object, classifier labels using a ClassifierConfig<\/code> object, and so on.<\/p>\n<\/a>
\n<\/figure>\n\n
Other Apple frameworks that use ML<\/h2>\n
Vision<\/h3>\n
\n
VNDetectHumanHandPoseRequest<\/code>)<\/li>\nVNDetectHumanBodyPoseRequest<\/code>)<\/li>\n<\/ul>\nCMSampleBuffer<\/code> objects straight from the camera with your request handlers.<\/p>\nVNStatefulRequest<\/code>, a subclass of VNImageBasedRequest<\/code> that builds up evidence over time<\/strong> of having detected the thing you\u2019re looking for. Unlike a typical VNImageBasedRequest<\/code>, you re-use a stateful request over multiple frames. It performs an analysis operation every N frames of the video.<\/p>\nVNObservation<\/code> object, which now has a timeRange<\/code> property that tells you when the observation started and stopped in the video.<\/p>\nVNStatefulRequest<\/code> directly \u2014 it\u2019s an abstract base class that is currently only subclassed by VNDetectTrajectoriesRequest<\/code> for doing trajectory detection. This can detect shapes moving along a parabolic path, like a ball being kicked or thrown<\/a>. (This appears to be the only built-in video task right now.)<\/p>\nVNVideoProcessor<\/code>. This object takes a URL to a local movie file and performs one or more Vision requests every N frames or seconds.<\/p>\nVNGenerateOpticalFlowRequest<\/code> that computes the direction each pixel is moving from one frame to the next (dense optical flow). The output is a VNPixelBufferObservation<\/code> containing a new image that has two 32-bit or 16-bit floats for each pixel.<\/p>\nVNDetectContoursRequest<\/code>, for detecting the outlines of objects<\/strong> in an image. These outlines are returned as vector paths. VNGeometryUtils<\/code> has helper functions for post-processing on the detected contours, such as simplifying them to basic geometric shapes.<\/p>\n\n
Natural Language Processing<\/h3>\n
\n
NLTagger<\/code> and NLModel<\/code> can now find multiple tags and predict their confidences. Previously, all you got was the highest scoring tag.<\/p>\n<\/li>\n