{"id":16582,"date":"2019-11-13T23:30:00","date_gmt":"2019-11-13T20:30:00","guid":{"rendered":"https:\/\/handoli.com\/index.php\/2019\/11\/13\/core-ml-and-combine\/"},"modified":"2019-11-13T23:30:00","modified_gmt":"2019-11-13T20:30:00","slug":"core-ml-and-combine","status":"publish","type":"post","link":"https:\/\/handoli.com\/index.php\/2019\/11\/13\/core-ml-and-combine\/","title":{"rendered":"Core ML and Combine"},"content":{"rendered":"<p>Over the weekend I was reading <a href=\"https:\/\/store.raywenderlich.com\/products\/combine-asynchronous-programming-with-swift\">Combine: Asynchronous Programming with Swift<\/a> from my friends at raywenderlich.com when it occured on me that Combine and Core ML might make a nice couple.<\/p>\n<p>Combine lets you build <strong>reactive event processing chains<\/strong>. Doing inference with a Core ML model can be one of the stages in such a chain, as this is nothing more than another data transformation operation.<\/p>\n<p>In this blog post I\u2019ll show a simple way to put Core ML inside a Combine event processing chain. You can find the code <a href=\"https:\/\/gist.github.com\/hollance\/a69e077429bf130ac18bc8ef61ed0e7c\">in this gist<\/a>.<\/p>\n<p>Keep in mind that I have a grand total of about 2 days experience writing Combine code, so there might be better ways to do this. \ud83d\ude05<\/p>\n<h2>Publishers, subscribers, operators<\/h2>\n<p>A typical processing chain for an image classifier looks something like this:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-combine\/ProcessingChain@2x.png\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-combine\/ProcessingChain.png\" alt=\"The processing chain for Core ML\" \/><\/a><br \/>\n<\/figure>\n<p>The first stage in this chain is a <strong>publisher<\/strong>, in Combine terminology, of images.<\/p>\n<p>The last stage of the chain is a <strong>subscriber<\/strong> that takes the classification results and displays them in the app.<\/p>\n<p>The in-between stages in the chain are <strong>operators<\/strong>. These subscribe to the previous publisher in the chain, transform the input they receive, and publish the results to the next subscriber in the chain.<\/p>\n<p>For an image classifier, you want an operator that performs inference on the input image using Core ML. This is followed by an operator that filters the results you\u2019re interested in from the model\u2019s prediction, for example the top-5 classes with the highest confidence scores.<\/p>\n<p>In this blog post you\u2019ll see how to create a custom Combine operator that lets you run any kind of Core ML model. There are actually several additional stages in the processing chain, but this is the general idea.<\/p>\n<div class=\"note\">\n<p><strong>Tip:<\/strong> Want to know more about how Combine works? <a href=\"https:\/\/store.raywenderlich.com\/products\/combine-asynchronous-programming-with-swift\">Read the book!<\/a> \ud83d\ude01 Combine isn\u2019t very well documented right now \u2014 and can be mind-boggling if you haven\u2019t done reactive programming before \u2014 but the team at raywenderlich.com did a great job at making the topic accessible.<\/p>\n<\/div>\n<h2>Get yourself a model<\/h2>\n<p>For this blog post I\u2019m assuming we\u2019re working with an image classifier \u2014 but the same technique works for any kind of model, even ones that don\u2019t use images.<\/p>\n<p>The model I\u2019m using in the sample code is SqueezeNet, the 16-bit version. You can <a href=\"https:\/\/docs-assets.developer.apple.com\/coreml\/models\/Image\/ImageClassification\/SqueezeNet\/SqueezeNetFP16.mlmodel\">download it<\/a> from <a href=\"https:\/\/developer.apple.com\/machine-learning\/models\/\">developer.apple.com\/machine-learning\/models\/<\/a>. Feel free to use any other image classifier model instead.<\/p>\n<p>Create an instance of the model using the Xcode-generated helper class:<\/p>\n<pre><code class=\"language-swift\">let squeezeNet = SqueezeNetFP16()\n<\/code><\/pre>\n<h2>It starts with an image\u2026<\/h2>\n<p>Because SqueezeNet is an image classifier, it makes sense to start the processing chain with an image object. Core ML likes to work with <code>CVPixelBuffer<\/code> objects, but let\u2019s say you have a <code>UIImage<\/code> instead.<\/p>\n<p>You can create a publisher for images as follows:<\/p>\n<pre><code class=\"language-swift\">let imagePublisher = PassthroughSubject&lt;UIImage, Never&gt;()\n<\/code><\/pre>\n<p>Now you can write <code>imagePublisher.send(someImage)<\/code> to send the <code>UIImage<\/code> object down the processing chain. Pretty easy.<\/p>\n<p>You could also use a different publisher here. For example, the <code>UIImage<\/code> could come from an event chain involving the <code>UIImagePickerController<\/code>, or from an event chain that downloaded the image using <code>NSURLSession<\/code>, or from any other publisher that outputs image objects.<\/p>\n<p>Or you could publish a <code>CGImage<\/code> or a <code>CVPixelBuffer<\/code> object instead of <code>UIImage<\/code>. And if your model expects a multi-array input, you\u2019d use a publisher that produces <code>MLMultiArray<\/code> objects.<\/p>\n<p>The whole point of Combine is that you can put all these different building blocks together in whatever specific way makes sense for your app.<\/p>\n<p>The goal in this blog post is to run the Core ML model on the <code>UIImage<\/code> coming from the <code>imagePublisher<\/code>. As you may know, Core ML does not directly accept <code>UIImage<\/code> objects as input so we\u2019ll have to convert it to a <code>CVPixelBuffer<\/code> somehow. That will be one of the stages in the processing chain.<\/p>\n<p>Before we get into that, let\u2019s look at how we\u2019re going to run Core ML using Combine in the first place.<\/p>\n<h2>The prediction operator<\/h2>\n<p>We need to create a Combine operator that lets us perform inference using a Core ML model and capture the output.<\/p>\n<p>We want to be able to write something like the following:<\/p>\n<pre><code class=\"language-swift\">imagePublisher\n  .prediction(model: squeezeNet)\n  .sink { value in print(value) }\n<\/code><\/pre>\n<p>The <code>prediction<\/code> operator is what runs the given Core ML model. This operator is connected to the <code>imagePublisher<\/code>, which is where it gets the input image from. In turn, it sends the prediction results to the <code>sink<\/code> subscriber, which displays them to the user \u2014 in this case, by printing the value to debug output.<\/p>\n<p>Every time you call <code>imagePublisher.send(someImage)<\/code>, this processing chain is automatically executed.<\/p>\n<p>We\u2019ll actually need to make the processing chain a bit more complicated than the above, but first let\u2019s look at how to implement this new <code>prediction<\/code> operator.<\/p>\n<p>Here is the code:<\/p>\n<pre><code class=\"language-swift\">extension Publisher where Self.Output: MLFeatureProvider {\n  public func prediction(model: MLModel) \n              -&gt; Publishers.Map&lt;Self, Result&lt;MLFeatureProvider, Error&gt;&gt; {\n    map { input in\n      do {\n        return .success(try model.prediction(from: input))\n      } catch {\n        return .failure(error)\n      }\n    }\n  }\n}\n\n<\/code><\/pre>\n<p>There are several ways to create a new Combine operator, but this is the easiest. You simply add a new function to the <code>Publisher<\/code> namespace. (An operator is both a publisher and a subscriber.)<\/p>\n<p>Combine really loves generics, and getting the type signature correct is half the battle of writing a new operator. Let\u2019s examine what all these types mean\u2026<\/p>\n<p>To make this a general-purpose operator that works with <em>any<\/em> Core ML model, we\u2019re using the <code>MLModel<\/code> API, not the helper class that Xcode automatically generates for you. (The generated helper code is specific to a particular model and won\u2019t work with a different model.)<\/p>\n<p>The key thing to know about the <code>MLModel<\/code> API is that the input data must be given by an <code>MLFeatureProvider<\/code> object, and the output data also gets placed inside an <code>MLFeatureProvider<\/code>.<\/p>\n<p>If you look at the automatically generated class for SqueezeNetFP16, you can see the input and output helper classes indeed do conform to <code>MLFeatureProvider<\/code>:<\/p>\n<pre><code class=\"language-swift\">class SqueezeNetFP16Input: MLFeatureProvider {\n  ...\n}\n\nclass SqueezeNetFP16Output: MLFeatureProvider {\n  ...\n}\n<\/code><\/pre>\n<p>Because the input data for the <code>MLModel<\/code> must be contained in some type that conforms to <code>MLFeatureProvider<\/code>, we\u2019ve defined the new <code>prediction<\/code> operator so that it only can be used with publishers whose output is an <code>MLFeatureProvider<\/code>:<\/p>\n<pre><code class=\"language-swift\">extension Publisher where Self.Output: MLFeatureProvider {\n<\/code><\/pre>\n<p>That\u2019s what the <code>where Self.Output: MLFeatureProvider<\/code> thing does. It tells Swift you can only call <code>prediction()<\/code> in a processing chain if the previous element in the chain outputs an <code>MLFeatureProvider<\/code>.<\/p>\n<p>The next line in the code was:<\/p>\n<pre><code class=\"language-swift\">  public func prediction(model: MLModel) \n              -&gt; Publishers.Map&lt;Self, Result&lt;MLFeatureProvider, Error&gt;&gt; {\n<\/code><\/pre>\n<p>A Combine operator is a function on a publisher that returns a new publisher object. It\u2019s important to get the return type correct.<\/p>\n<p>We\u2019re simply reusing an existing publisher type, <code>Publishers.Map<\/code>, that is the result of performing a <code>map<\/code> operation on the processing chain. After all, doing inference with a Core ML model is really nothing more than a transformation on some data \u2014 and that\u2019s exactly what <code>map<\/code> does too.<\/p>\n<p>The <code>Publishers.Map<\/code> object takes two generic types:<\/p>\n<ol>\n<li>\n<p>First, the type of the upstream publisher, which is <code>Self<\/code>. The upstream publisher is the one whose output you\u2019re running <code>prediction()<\/code> on.<\/p>\n<\/li>\n<li>\n<p>The second type is <code>Result&lt;MLFeatureProvider, Error&gt;<\/code>. This describes the output from the Core ML model.<\/p>\n<\/li>\n<\/ol>\n<p>If all goes well, the Core ML output is a new <code>MLFeatureProvider<\/code> object that contains whatever the model computed. However, things don\u2019t always go well. In that case, Core ML throws an error.<\/p>\n<p>There are different ways in which you can handle errors with Combine. Normally, if an error is thrown in a Combine processing chain, the subscription is \u201ccompleted\u201d in a failure mode and the subscriber no longer receives any events. I don\u2019t really like that for Core ML errors, so another way to handle potential error situations is to return a <code>Result<\/code> object.<\/p>\n<p>Actually running the Core ML model is done by this code:<\/p>\n<pre><code class=\"language-swift\">    map { input in\n      do {\n        return .success(try model.prediction(from: input))\n      } catch {\n        return .failure(error)\n      }\n    }\n<\/code><\/pre>\n<p>Here, <code>map<\/code> is the Combine operator that converts the <code>input<\/code>, which is the <code>MLFeatureProvider<\/code> object provided by the upstream publisher, into a <code>Result<\/code> object that will be sent downstream.<\/p>\n<p>We simply run the usual <code>MLModel<\/code> <code>prediction()<\/code> method and either return a <code>Result.success<\/code> with the output <code>MLFeatureProvider<\/code> or a <code>Result.failure<\/code> with the thrown <code>Error<\/code> object.<\/p>\n<p>It\u2019s now up to any other stages in the processing chain to figure out how to handle the <code>Result<\/code> object, and to deal with the error in case it\u2019s a <code>.failure<\/code>.<\/p>\n<p>If you always want to silently ignore errors, you can write the <code>prediction<\/code> operator using <code>compactMap<\/code> instead of <code>map<\/code>, like so:<\/p>\n<pre><code class=\"language-swift\">public func prediction(model: MLModel) \n            -&gt; Publishers.CompactMap&lt;Self, MLFeatureProvider&gt; {\n  compactMap { input in try? model.prediction(from: input) }\n}\n<\/code><\/pre>\n<p>Now if Core ML gives an error, <code>try?<\/code> returns <code>nil<\/code> and <code>compactMap<\/code> will ignore such results \u2014 they don\u2019t get pushed any further down the processing chain. Note that the return type is different now too. We\u2019re returning a <code>Publishers.CompactMap<\/code> object; and <code>Result<\/code> isn\u2019t needed because things can never go wrong.<\/p>\n<p>It\u2019s also possible to use <code>tryMap<\/code> to complete the subscription upon the first error:<\/p>\n<pre><code class=\"language-swift\">public func prediction(model: MLModel) \n            -&gt; Publishers.TryMap&lt;Self, MLFeatureProvider?&gt; {\n  tryMap { input in try model.prediction(from: input) }\n}\n<\/code><\/pre>\n<p>In fact, you can have all three of these versions of <code>prediction()<\/code> in the code base. Swift can tell them apart based on the type of the input used in the next stage of the chain. For example, if the next stage expects an <code>MLFeatureProvider?<\/code> object, it\u2019s obvious it should use the <code>tryMap<\/code> version of <code>prediction()<\/code>.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> If you get an error message, \u201cAmbiguous use of \u2018prediction(model:)\u2018\u201c, it means Swift has trouble figuring out which version of the operator to use. You may need to provide explicit type annotations to disambiguate.<\/p>\n<\/div>\n<h2>Building the chain<\/h2>\n<p>At this point, we\u2019ve got two parts of the processing chain:<\/p>\n<ol>\n<li>the publisher that provides <code>UIImage<\/code> objects<\/li>\n<li>the operator that can run inference using a Core ML model<\/li>\n<\/ol>\n<p>But we can\u2019t connect these two things together just yet. This won\u2019t work:<\/p>\n<pre><code class=\"language-swift\">imagePublisher\n  .prediction(model: squeezeNet.model)\n  .sink(receiveValue: { value in\n    print(value)\n  })\n<\/code><\/pre>\n<p>The problem is that <code>imagePublisher<\/code> produces <code>UIImage<\/code> objects but, as you\u2019ve seen in the previous section, <code>prediction()<\/code> expects to receive an <code>MLFeatureValue<\/code> (because of the generics constraint we put on <code>Self.Output<\/code>).<\/p>\n<p>If you tried to type in the above code, Xcode\u2019s autocomplete does not suggest <code>prediction<\/code> as a valid thing after <code>imagePublisher.<\/code>, which is a good hint that the two cannot be directly connected.<\/p>\n<p>What we need is some way to convert the <code>UIImage<\/code> into an <code>MLFeatureProvider<\/code> object.<\/p>\n<p>This is not limited to image classifiers, by the way. Since we\u2019re using the <code>MLModel<\/code> API to run Core ML, all input data must be put into <code>MLFeatureProvider<\/code>, also if it\u2019s an <code>MLMultiArray<\/code> or integers or strings or anything else.<\/p>\n<p>Each piece of input data must first be converted to <code>MLFeatureValue<\/code>, and all the <code>MLFeatureValue<\/code>s for a given prediction request need to be placed into the same <code>MLFeatureProvider<\/code> object.<\/p>\n<p>Also, recall that Core ML wants a <code>CVPixelBuffer<\/code> object, not a <code>UIImage<\/code>. <a href=\"https:\/\/github.com\/hollance\/CoreMLHelpers\">There are ways<\/a> to convert from <code>UIImage<\/code> to <code>CVPixelBuffer<\/code>, but as of iOS 13, you can directly use a handy API on <code>MLFeatureValue<\/code>.<\/p>\n<p>Here\u2019s how to do it:<\/p>\n<pre><code class=\"language-swift\">let inputName = \"image\"\nlet imageConstraint = squeezeNet.model.modelDescription\n                                .inputDescriptionsByName[inputName]!\n                                .imageConstraint!\n\nlet imageOptions: [MLFeatureValue.ImageOption: Any] = [\n  .cropAndScale: VNImageCropAndScaleOption.scaleFill.rawValue\n]\n\nimagePublisher\n  .compactMap { image in                            \/\/ 1\n    try? MLFeatureValue(cgImage: image.cgImage!,\n                        orientation: .up,\n                        constraint: imageConstraint,\n                        options: imageOptions)\n  }\n  .compactMap { featureValue in                     \/\/ 2\n    try? MLDictionaryFeatureProvider(dictionary: [inputName: featureValue])\n  }\n  .prediction(model: squeezeNet.model)\n<\/code><\/pre>\n<p>The conversion from <code>UIImage<\/code> to <code>MLFeatureProvider<\/code> is split up into two steps:<\/p>\n<ol>\n<li>\n<p>First we load the image into an <code>MLFeatureValue<\/code>. This API takes the <code>UIImage<\/code> \u2014 or actually its <code>CGImage<\/code> \u2014 and scales it to the image dimensions the model expects. It can also rotate the image if it\u2019s not upright. The scaled and rotated image is stored in an <code>MLFeatureValue<\/code> object. For each input in your model, you need to provide such an <code>MLFeatureValue<\/code>.<\/p>\n<\/li>\n<li>\n<p>All the feature value objects for your model\u2019s inputs are placed into a single <code>MLFeatureProvider<\/code>. This is done in the second step. We use a concrete implementation, <code>MLDictionaryFeatureProvider<\/code>, that takes a dictionary that maps the <code>MLFeatureValue<\/code> objects to the model\u2019s named inputs. There\u2019s only one input in SqueezeNet, named <code>\"image\"<\/code>, but if your model has multiple inputs you\u2019d specify them all in this dictionary.<\/p>\n<\/li>\n<\/ol>\n<p>For both of these steps we\u2019re using the <code>compactMap<\/code> operator, so that if the <code>try?<\/code> fails and returns <code>nil<\/code>, the rest of the chain silently ignores the error. You can use <code>tryMap<\/code> instead in case you want such errors to propagate down the chain.<\/p>\n<p>Now that the <code>UIImage<\/code> is wrapped in an <code>MLFeatureProvider<\/code>, you can finally call <code>prediction()<\/code> to perform inference.<\/p>\n<p>Note that you can also replace the second <code>compactMap<\/code> with the following:<\/p>\n<pre><code class=\"language-swift\">  .map { featureValue in\n    SqueezeNetFP16Input(image: featureValue.imageBufferValue!)\n  }\n<\/code><\/pre>\n<p>This uses the Xcode-generated helper class, <code>SqueezeNetFP16Input<\/code>, which as you\u2019ll recall implements <code>MLFeatureProvider<\/code>. This can\u2019t go wrong, so you can use <code>map<\/code> here instead of <code>compactMap<\/code>. (The helper class actually expects the image as a <code>CVPixelBuffer<\/code>, which you can get from the <code>MLFeatureValue<\/code> object using the <code>imageBufferValue<\/code> property.)<\/p>\n<p>If your model uses an <code>MLMultiArray<\/code> as input, or if your images aren\u2019t <code>UIImage<\/code>, you\u2019ll naturally need to change how the input data gets mapped. You can use any Combine operators here, just make sure that the input to <code>prediction()<\/code> is an <code>MLFeatureProvider<\/code> of some kind.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> When you call <code>prediction()<\/code>, you need to pass in an <code>MLModel<\/code> instance. You can\u2019t use the <code>squeezeNet<\/code> variable, which is an instance of the auto-generated helper class <code>SqueezeNetFP16<\/code>. Instead, you need to write <code>squeezeNet.model<\/code> to get the <code>MLModel<\/code> object.<\/p>\n<\/div>\n<h2>Processing the results<\/h2>\n<p>Depending on which version of <code>prediction()<\/code> you use, its return value is either an <code>MLFeatureProvider<\/code>, the optional version of that, or a <code>Result<\/code> object.<\/p>\n<p>Actually, at this point in time it could still be any one of these\u2026 Swift doesn\u2019t have enough information yet to decide which one to use. The next stage in the chain will determine this.<\/p>\n<p>For example, if you write the following,<\/p>\n<pre><code class=\"language-swift\">imagePublisher\n  \/* conversion from UIImage to MLFeatureProvider omitted *\/\n  .prediction(model: squeezeNet.model)\n  .map { result -&gt; String in\n    switch result {\n      case .success(let featureProvider):\n        return featureProvider.featureValue(for: \"classLabel\")?\n                              .stringValue ?? \"error\"\n      case .failure(let error):\n        return error.localizedDescription\n    }\n  }\n  .sink(receiveValue: { value in print(value) })\n<\/code><\/pre>\n<p>Swift will use the <code>prediction()<\/code> that returns a <code>Result<\/code> object, because the switch statement obviously expects a <code>Result<\/code> here. Inside this <code>map<\/code> operation, we convert the prediction result into a string. The <code>sink<\/code> subscriber at the very end of the chain simply prints this string to the debug output.<\/p>\n<p>The above works fine but it\u2019s still a bit messy, especially the code needed to read the results from the <code>MLFeatureProvider<\/code> object:<\/p>\n<pre><code class=\"language-swift\">featureProvider.featureValue(for: \"classLabel\")?.stringValue ?? \"error\"\n<\/code><\/pre>\n<p>Yuck. The whole reason Xcode generates the helper classes is to hide this nastiness from you, so let\u2019s use those helpers as much as possible:<\/p>\n<pre><code class=\"language-swift\">imagePublisher\n  \/* conversion from UIImage to MLFeatureProvider omitted *\/\n  .prediction(model: squeezeNet.model)\n  .compactMap { try? $0.get() }                  \/\/ 1\n  .map { SqueezeNetFP16Output(features: $0) }    \/\/ 2\n  .map { $0.classLabel }                         \/\/ 3\n  .sink(receiveValue: { value in print(value) })\n<\/code><\/pre>\n<p>Here\u2019s how this works:<\/p>\n<ol>\n<li>\n<p>The <code>.compactMap<\/code> step calls <code>get()<\/code> on the <code>Result<\/code> object. If the result was successful, this returns the <code>MLFeatureProvider<\/code> object. This does the same as the first case in the switch statement above.<\/p>\n<p>If the <code>Result<\/code> was not successful, the <code>try?<\/code> returns nil and <code>compactMap<\/code> silently ignores the error. As before, you can use <code>tryMap<\/code> here too.<\/p>\n<\/li>\n<li>\n<p>The first <code>map<\/code> puts the <code>MLFeatureProvider<\/code> inside the generated helper class, <code>SqueezeNetFP16Output<\/code>. This makes the next step more convenient.<\/p>\n<\/li>\n<li>\n<p>Here you grab the <code>classLabel<\/code> property from the helper class. This property contains the label of the class with the highest predicted probability. (If you\u2019re not using SqueezeNet, this property may have a different name in your model.)<\/p>\n<\/li>\n<\/ol>\n<p>And finally, we use the sink subscriber to print the string to the debug output.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> You can of course combine the two <code>map<\/code> operations into one:<\/p>\n<p><code>.map { SqueezeNetFP16Output(features: $0).classLabel }<\/code><\/p>\n<p>The reason I used two different steps is that some models have more than one output, and you may only want to do the conversion to the helper class just once.<\/p>\n<\/div>\n<p>It\u2019s easy enough to change the result processing: just replace the <code>map<\/code> operation with something else.<\/p>\n<p>For example, if you wanted to get the dictionary of predicted probabilities for all classes instead of just the highest probability class label, you\u2019d simply grab the <code>$0.classLabelProbs<\/code> property. Now the sink receives a dictionary of <code>[String: Double]<\/code> values instead of a string.<\/p>\n<p>You may also want to do some additional post-processing on the prediction results. For example, here\u2019s how you\u2019d get the top-5 classes from the dictionary:<\/p>\n<pre><code class=\"language-swift\">.map {\n  $0.classLabelProbs.sorted { $0.value &gt; $1.value }.prefix(5)\n}\n<\/code><\/pre>\n<p>For a model <a href=\"https:\/\/machinethink.net\/blog\/object-detection\/\">such as SSD<\/a> that predicts offsets to anchor boxes, you could do the bounding box decoding here, and so on\u2026<\/p>\n<h2>The complete chain<\/h2>\n<p>Here\u2019s what the full source code looks like:<\/p>\n<pre><code class=\"language-swift\">class MyClass {\n  let squeezeNet = SqueezeNetFP16()\n  let imagePublisher = PassthroughSubject&lt;UIImage, Never&gt;()\n  var subscriptions = Set&lt;AnyCancellable&gt;()\n\n  func setUpProcessingChain() {\n    let imageConstraint = squeezeNet.model.modelDescription\n                                    .inputDescriptionsByName[\"image\"]!\n                                    .imageConstraint!\n\n    let imageOptions: [MLFeatureValue.ImageOption: Any] = [\n      .cropAndScale: VNImageCropAndScaleOption.scaleFill.rawValue\n    ]\n\n    imagePublisher\n      .compactMap { image in\n        try? MLFeatureValue(cgImage: image.cgImage!,\n                            orientation: .up,\n                            constraint: imageConstraint,\n                            options: imageOptions)\n      }\n      .map { featureValue in\n        SqueezeNetFP16Input(image: featureValue.imageBufferValue!)\n      }\n      .prediction(model: squeezeNet.model)\n      .compactMap { try? $0.get() }\n      .map { SqueezeNetFP16Output(features: $0) }\n      .map { $0.classLabelProbs.sorted { $0.value &gt; $1.value }.prefix(5) } \n      .sink(receiveCompletion: { completion in\n        print(\"Received completion event\")\n      }, receiveValue: { value in\n        print(\"Received value:\", value)\n        \/* do something with the predicted value here *\/\n      })\n      .store(in: &amp;subscriptions)\n  }\n\n  func performInference(image: UIImage) {\n    imagePublisher.send(image)\n  }\n}\n<\/code><\/pre>\n<p><code>setUpProcessingChain()<\/code> should be called just once to set everything up. And now you have a Combine processing chain that, every time you send it a <code>UIImage<\/code> object with <code>imagePublisher.send(image)<\/code>, will automatically run a Core ML model on that image and process the results. Pretty cool!<\/p>\n<h2>Conclusion<\/h2>\n<p>Should you use this technique? It certainly makes sense if you\u2019re using Combine elsewhere in your app.<\/p>\n<p>Since the <code>prediction()<\/code> operator works like any Combine operator \u2014 it\u2019s just a convenient wrapper around <code>map<\/code> \u2014 you can put it into any of your Combine processing chains.<\/p>\n<p>For even more convenience, you could even implement conversion from <code>UIImage<\/code> (or whatever) to <code>MLFeatureProvider<\/code> as another operator:<\/p>\n<pre><code class=\"language-swift\">imagePublisher\n  .toFeatureProvider(image: someImage)\n  .prediction(model: ...)\n  .postprocessPredictionResults()\n  .sink { ... }\n<\/code><\/pre>\n<p>As I write this, I haven\u2019t used Combine much yet, and definitely not in a real app, but it looks like an interesting way to do reactive programming on iOS devices. And now it works with Core ML too!<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> This blog post showed only a very simple implementation of <code>prediction()<\/code> that blocks the current thread. To perform the Core ML request asynchronously, it\u2019s probably a good idea to write a custom <code>Publisher<\/code> class. But that\u2019s a topic for another day.<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"<p>Over the weekend I was reading Combine: Asynchronous Programming with Swift from my friends at raywenderlich.com when it occured on me that Combine and Core ML might make a nice couple. Combine lets you build reactive event processing chains. Doing inference with a Core ML model can be one of the stages in such a [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":16583,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rop_custom_images_group":[],"rop_custom_messages_group":[],"rop_publish_now":"initial","rop_publish_now_accounts":[],"rop_publish_now_history":[],"rop_publish_now_status":"pending","footnotes":""},"categories":[1],"tags":[],"class_list":["post-16582","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-explore"],"_links":{"self":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/posts\/16582","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/comments?post=16582"}],"version-history":[{"count":0,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/posts\/16582\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/media\/16583"}],"wp:attachment":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/media?parent=16582"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/categories?post=16582"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/tags?post=16582"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}