{"id":10201,"date":"2019-12-02T00:00:00","date_gmt":"2019-12-01T21:00:00","guid":{"rendered":"https:\/\/handoli.com\/index.php\/2019\/12\/02\/on-device-training-with-core-ml-part-4\/"},"modified":"2019-12-02T00:00:00","modified_gmt":"2019-12-01T21:00:00","slug":"on-device-training-with-core-ml-part-4","status":"publish","type":"post","link":"https:\/\/handoli.com\/index.php\/2019\/12\/02\/on-device-training-with-core-ml-part-4\/","title":{"rendered":"On-device training with Core ML &#8211; part 4"},"content":{"rendered":"<p>In this series of blog posts we\u2019re taking a deep dive into the new on-device <strong>model personalization<\/strong> features from Core ML 3. I\u2019ll show how to create a customizable image classifier using k-Nearest Neighbors as well as a deep neural network, right from inside an iOS app.<\/p>\n<p>This is the last part of a four-part series:<\/p>\n<ol>\n<li><a href=\"https:\/\/machinethink.net\/blog\/coreml-training-part1\">Introduction to on-device training<\/a><\/li>\n<li><a href=\"https:\/\/machinethink.net\/blog\/coreml-training-part2\">Rock, Paper, Scissors (Lizard? Spock?)<\/a><\/li>\n<li><a href=\"https:\/\/machinethink.net\/blog\/coreml-training-part3\">k-Nearest Neighbors<\/a><\/li>\n<li><strong>Training a Neural Network<\/strong><\/li>\n<\/ol>\n<p>Follow along with the <a href=\"https:\/\/github.com\/hollance\/coreml-training\">source code on GitHub<\/a>.<\/p>\n<p>In this last installment, we\u2019ll finally bring out the big guns: a deep neural network.<\/p>\n<h2>Training a neural network<\/h2>\n<p>Even though Core ML 3 allows you to train advanced neural networks on the iPhone, I should point out that <strong>k-Nearest Neighbors is often the better choice<\/strong>!<\/p>\n<p>Developers should probably use k-NN for many, if not most, of their on-device personalization tasks. Usually in combination with a neural network that acts as a feature extractor, like we did in <a href=\"https:\/\/machinethink.net\/blog\/coreml-training-part3\/\">part 3<\/a>.<\/p>\n<p>The advantages of k-NN are that it\u2019s simple and fast to train, only requires a few training examples per class, and it can handle any number of classes that can be defined by the user.<\/p>\n<p>A neural network is usually quite complex, is slower to train because it needs many training examples, and the architecture is set in stone: if your neural network can only predict 3 classes, you cannot add a 4th class later.<\/p>\n<p>If it is at all possible for the number of classes to change at runtime, you already need to take that into account when you design the architecture of the neural net and make its classification layer large enough.<\/p>\n<p>For example, the dataset we\u2019ve been using only has three classes \u2014 rock \u270a, paper \u270b, scissors \u270c\ufe0f \u2014 but we allow the user to add new gestures. To accomodate this, the network\u2019s classification layer actually contains 10 instead of 3 neurons so that it can predict up to 10 classes, even if they\u2019re not all used by the app.<\/p>\n<p><strong>So why use a neural network?<\/strong> Well, it\u2019s way more powerful than k-NN and can give better results.<\/p>\n<ul>\n<li>\n<p>k-NN doesn\u2019t really \u201clearn\u201d anything \u2014 it just memorizes \u2014 and so it might not generalize as well as a model type that is truly capable of learning.<\/p>\n<\/li>\n<li>\n<p>k-NN models are also not very robust against noisy data, or when there are outliers, or when the data contains irrelevant features. Neural networks are much better at that sort of thing.<\/p>\n<\/li>\n<li>\n<p>Even though k-NN in general can also be used for regression, in Core ML it only does classification. If your model is not a classifier, you\u2019ll need to use a neural network. (But note that you can\u2019t just train <em>any<\/em> neural network in Core ML \u2014 there are still many limitations.)<\/p>\n<\/li>\n<\/ul>\n<p>My recommendation is to start out with a k-NN model first if you\u2019re interested in doing on-device training. If this works well enough for your kind of data, great. I\u2019d only switch to a full neural network if k-NN\u2019s predictions aren\u2019t good enough.<\/p>\n<h3>How does training work again?<\/h3>\n<p>You probably already know how neural networks are trained but here\u2019s a quick refresher.<\/p>\n<p>Assuming that you have a large-enough dataset of images and their labels, you present all the training examples to the model one after the other.<\/p>\n<ol>\n<li>The model makes predictions (the forward pass),<\/li>\n<li>the <strong>loss function<\/strong> computes how wrong these predictions are,<\/li>\n<li>based on how wrong the model is, using backpropagation the <strong>optimizer<\/strong> slightly adjust the learned parameters (the backward pass),<\/li>\n<li>so that the next time the model\u2019s predictions will be slightly better.<\/li>\n<\/ol>\n<p>Each run through the dataset is called an <strong>epoch<\/strong>. This is repeated tens or even hundreds of times, and on every one of these epochs the model becomes a bit better at making predictions.<\/p>\n<p>Usually you don\u2019t train one image at a time but using <strong>mini-batches<\/strong> of, say, 32 images. Ideally you\u2019d compute the loss over <em>all<\/em> the images at once but this is impractical as it would require way too much RAM. So instead, you approximate the loss for the entire dataset with the loss over a much smaller mini-batch.<\/p>\n<p>Using mini-batches works better than using single images and is a little more efficient too.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> Exactly how large mini-batches should be is a matter of some debate. A typical batch size is 32. In practice it depends on how much RAM you have to spare, and on mobile RAM is often quite limited. That\u2019s why in the demo app we\u2019ll use a mini-batch size of 8. But be aware that on models that use batch normalization layers, the mini-batch size should not be too small.<\/p>\n<\/div>\n<p>Core ML takes care of pretty much all this stuff, so you don\u2019t need to know how backpropagation or loss functions or optimizers work. But you do need to set up the <strong>hyperparameters<\/strong> for the training process properly.<\/p>\n<h2>The model architecture<\/h2>\n<p>The neural network we\u2019ll be using is a <strong>SqueezeNet v1.1<\/strong> image classifier. I chose SqueezeNet for this blog post because it\u2019s quite small and also because it\u2019s what you get when you train an image classifier with <a href=\"https:\/\/github.com\/apple\/turicreate\">Turi Create<\/a>.<\/p>\n<p>Yep, it\u2019s the exact same model that <a href=\"https:\/\/machinethink.net\/blog\/coreml-training-part3\/\">we\u2019ve used with k-NN<\/a>, except this time we won\u2019t strip off the classifier layer.<\/p>\n<p>Just like Turi Create and Create ML, our app will be using <strong>transfer learning<\/strong>. Instead of training the entire model from scratch \u2014 which I think we agreed is a <a href=\"https:\/\/machinethink.net\/blog\/coreml-training-part1\/\">bad idea on mobile<\/a> \u2014 only the last layer of SqueezeNet will be (re-)trained. This is the layer that does the actual classification work.<\/p>\n<p>The rest of SqueezeNet was pre-trained on the ImageNet dataset (photos of 1000 types of objects). Just like with the k-NN model, this part of SqueezeNet acts as a feature extractor.<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-training\/ClassifierLayer@2x.png\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-training\/ClassifierLayer.png\" alt=\"We train the last innerProduct layer\"\/><\/a><br \/>\n<\/figure>\n<p>Unlike with k-NN, you can\u2019t easily add new classes to the neural network. So you need to decide beforehand how many classes your model will have. In our case, it\u2019s 10 classes max.<\/p>\n<p>The layer that we\u2019ll be training is a <strong>fully-connected layer<\/strong>, also known as an <strong>inner product<\/strong> or <strong>affine<\/strong> layer. It has 1000 input neurons and 10 output neurons (because 10 classes). Our neural network will therefore need to learn 1000 \u00d7 10 = 10,000 weights plus 10 bias values.<\/p>\n<p>The illustration above also shows other layers: softmax, flatten, pooling, activation. These layers do not have any learnable parameters and therefore do not need to be trained.<\/p>\n<h3>A quick aside about logistic regression<\/h3>\n<p>If you only train the very last layer like we\u2019re going to be doing here, you\u2019re basically just training a <strong>logistic regression<\/strong> model on top of a feature extractor.<\/p>\n<p>Logistic regression can be trained very quickly using an optimizer such as L-BFGS. In fact, that is what Turi Create does for its image classifiers. That is much faster than training a neural network.<\/p>\n<p>Core ML currently doesn\u2019t support training logistic regression but you could roll your own. So why are we using a neural network instead?<\/p>\n<p>One downside of logistic regression is that it\u2019s harder to apply data augmentation. Logistic regression must always be trained on the entire dataset at once, so you\u2019d need to precompute all the augmented images and their feature vectors (this is what Create ML does). Neural networks are trained on small batches of data at a time, allowing us to do data augmentation on-the-fly.<\/p>\n<p>Also, with a neural network we\u2019re not limited to just training the last layer: we can also train other layers in order to fine-tune parts of the feature extractor on the user\u2019s data too. This usually gives much better results.<\/p>\n<p>That said, using the neural network as a feature extractor, followed by logistic regression, is definitely a plausible alternative for doing on-device training. You\u2019ll just have to implement the training logic for this yourself.<\/p>\n<h2>Creating the mlmodel<\/h2>\n<p>In order to train on the device we need to start out from a previously trained mlmodel. Two of these are included in the app:<\/p>\n<ul>\n<li><strong>HandsTuri.mlmodel<\/strong>, made by Turi Create<\/li>\n<li><strong>HandsEmpty.mlmodel<\/strong>, the same model but with an untrained classifier layer<\/li>\n<\/ul>\n<p>Both these models are derived from <strong>TuriOriginal.mlmodel<\/strong>. As you may recall from <a href=\"https:\/\/machinethink.net\/blog\/coreml-training-part2\/\">part 2<\/a>, as a proof-of-concept I first trained a SqueezeNet model using Turi Create. This model, which is TuriOriginal.mlmodel, gets 80% accuracy on the test set. (See the <strong>Scripts\/TuriCreate.ipynb<\/strong> notebook for more details.)<\/p>\n<p>The starter models are created by the Python script <strong>Scripts\/make_nn.py<\/strong>. It first loads TuriOriginal.mlmodel, makes it updatable and saves this as HandsTuri. Next, it resets the weights of the classifier layer and saves that as HandsEmpty.mlmodel.<\/p>\n<p>Let\u2019s take a look at how <strong>make_nn.py<\/strong> works.<\/p>\n<pre><code class=\"language-python\">import coremltools\nmodel = coremltools.models.MLModel(\"TuriOriginal.mlmodel\")\n\nspec = model._spec\nlayer = spec.neuralNetworkClassifier.layers[-2]\n\nnum_classes = 10\nlayer.innerProduct.outputChannels = num_classes\n\nweights = np.zeros((num_classes - 3) * 1000)\nbiases = np.zeros(num_classes - 3)\nlabels = [\"user\" + str(i) for i in range(num_classes - 3)]\n\nlayer.innerProduct.weights.floatValue.extend(weights)\nlayer.innerProduct.bias.floatValue.extend(biases)\nspec.neuralNetworkClassifier.stringClassLabels.vector.extend(labels)\n<\/code><\/pre>\n<p>This loads the original mlmodel and makes a few changes to the classifier layer. In Turi Create, I only trained the model on 3 classes. That means the fully-connected layer also has 3 neurons. But because we want to user to add their own classes, we have to give this layer more neurons.<\/p>\n<p>For this app, I decided there can be at most 10 gestures. 10 possible classes is 10 output neurons, or <code>outputChannels<\/code> as Core ML calls them. (k-NN models don\u2019t have this kind of limit on the number of classes.)<\/p>\n<p>More neurons also means the layer needs to have more weights. The above code adds these extra weights to the layer. Because we don\u2019t want these to interfere with the existing learned parameters of this model, the new weights are set to 0.<\/p>\n<p>We must also add placeholder names for the new classes, which I named <code>user0<\/code>, <code>user1<\/code>, and so on.<\/p>\n<p>OK, that was a bit of model surgery to make the neural network capable of learning more than three classes. Now let\u2019s make this model updatable. First, put the model into a <code>NeuralNetworkBuilder<\/code>:<\/p>\n<pre><code>from coremltools.models.neural_network import NeuralNetworkBuilder\nbuilder = NeuralNetworkBuilder(spec=model._spec)\n<\/code><\/pre>\n<p>Previously, <code>NeuralNetworkBuilder<\/code> could only be used to create new models, but as of coremltools 3.0 you can also use it to make changes to existing models, which is less effort than messing around with the protobuf objects yourself.<\/p>\n<p>To make an mlmodel trainable on the device,<\/p>\n<ul>\n<li>its <code>isUpdatable<\/code> property must be set to true<\/li>\n<li>any neural network layers that can be trained must also have their <code>isUpdatable<\/code> property set to true<\/li>\n<li>the mlmodel file must have a specification version of at least 4<\/li>\n<li>the neural network must include a loss function<\/li>\n<li>the model must have training inputs in addition to the regular inputs<\/li>\n<li>the model must have an optimizer<\/li>\n<li>the mlmodel file must contain default values for hyperparameters such as the mini-batch size and the learning rate<\/li>\n<\/ul>\n<p>The <code>NeuralNetworkBuilder<\/code> makes it easy to do all of these things.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> You might also be able to use the Core ML converters to make your models updatable. Currently, only the Keras converter supports this. With the option <code>respect_trainable=True<\/code>, any layers that were trainable in the original model also become trainable in the Core ML model. (Unfortunately, the Keras converter only does half a job and you still need to add the <em>training inputs<\/em> by hand.) But because we\u2019re starting from a Turi Create mlmodel, not a Keras model, we have to make it trainable by hand using <code>NeuralNetworkBuilder<\/code>.<\/p>\n<\/div>\n<h3>Choose which layers are updatable<\/h3>\n<p>To mark the model and any layers as being updatable, write the following:<\/p>\n<pre><code class=\"language-python\">builder.make_updatable([\"fullyconnected0\"])\n<\/code><\/pre>\n<p>Here, we\u2019re saying that <code>\"fullyconnected0\"<\/code>, which is the final classification layer from our model, will be trainable.<\/p>\n<p>For this simple demo app we\u2019re only training the last layer of the neural network, but of course it\u2019s possible to train more than one layer. Simply pass in a list of layer names, for example <code>[\"fullyconnected0\", \"conv10\"]<\/code>.<\/p>\n<p>Currently, only <strong>fully-connected<\/strong> (also known as inner-product or affine) layers and <strong>convolution layers<\/strong> can be made trainable. But, as I mentioned, a lot of layer types don\u2019t need training \u2014 such as softmax, flatten, pooling, and activation layers. You don\u2019t need to specify those when you call <code>builder.make_updatable()<\/code>.<\/p>\n<p>There is another important limitation: Core ML cannot yet backpropagate through all layer types. Between <code>\"conv10\"<\/code> and the preceding convolution layers is a <em>concat<\/em> layer. Unfortunately, Core ML cannot backprop through such a concat layer, which means that training <code>\"fullyconnected0\"<\/code> and <code>\"conv10\"<\/code> is the best you can do for SqueezeNet. The other convolution layers are not reachable.<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-training\/UnreachableLayers@2x.png\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-training\/UnreachableLayers.png\" alt=\"Backprop can&apos;t reach the layers beyond the concat layer\"\/><\/a><br \/>\n<\/figure>\n<p><code>make_updatable()<\/code> also automatically sets the <code>specificationVersion<\/code> of the mlmodel to 4. This version number is important, so that apps won\u2019t try to use updatable models with older iOS versions. (The downside is that you can\u2019t use a model that is updatable on iOS 12 or earlier, even if you\u2019re not using any of the training stuff.)<\/p>\n<h3>The loss function and training inputs<\/h3>\n<p>Next, we\u2019ll need to add a <strong>loss function<\/strong> to the neural network.<\/p>\n<p>Core ML currently offers two loss functions:<\/p>\n<ul>\n<li>cross-entropy loss<\/li>\n<li>mean squared error loss<\/li>\n<\/ul>\n<p>You can\u2019t define your own loss functions, so this limits neural networks to multi-class classification and basic regression tasks.<\/p>\n<p>Since our model is a classifier, we need to use <strong>cross-entropy loss<\/strong>:<\/p>\n<pre><code class=\"language-python\">builder.set_categorical_cross_entropy_loss(name=\"lossLayer\",\n                                           input=\"labelProbability\")\n<\/code><\/pre>\n<p>The <code>input<\/code> argument is the name of the output from the softmax layer. Because our model is a standard Core ML classifier, it has the following outputs:<\/p>\n<ul>\n<li><code>\"label\"<\/code>: the class label of the best prediction<\/li>\n<li><code>\"labelProbability\"<\/code>: a dictionary with the predicted probabilities for all classes<\/li>\n<\/ul>\n<p>The cross-entropy loss layer needs to be connected to the dictionary output, <code>\"labelProbability\"<\/code>. The loss layer reads the predicted probabilities from this output and compares those to the training example\u2019s true label in order to compute the loss.<\/p>\n<p>Speaking of the true labels\u2026 where do those come from? Well, besides adding a loss layer, <code>set_categorical_cross_entropy_loss()<\/code> also adds so-called <strong>training inputs<\/strong> to your model.<\/p>\n<p>Before, our SqueezeNet model had the following input:<\/p>\n<ul>\n<li><code>\"image\"<\/code>: a 227&#215;227 RGB image<\/li>\n<\/ul>\n<p><code>set_categorical_cross_entropy_loss()<\/code> adds two new inputs:<\/p>\n<ul>\n<li><code>\"image\"<\/code>: a 227&#215;227 RGB image<\/li>\n<li><code>\"label\"<\/code>: a string<\/li>\n<\/ul>\n<p>These are called the <strong>training inputs<\/strong> because Core ML uses these during training instead of the regular inputs. Note that <code>\"image\"<\/code> appears in both the regular inputs and the training inputs \u2014 after all, you\u2019ll also need the image during training.<\/p>\n<p>The new addition is the input for the <strong>true label<\/strong>, named simply <code>\"label\"<\/code>.<\/p>\n<p>At this point \u2014 if you\u2019re curious \u2014 you can look at the loss layer by doing <code>print(builder.spec.neuralNetworkClassifier.updateParams)<\/code>. This prints the following:<\/p>\n<pre><code class=\"language-nohighlight\">lossLayers {\n  name: \"lossLayer\"\n  categoricalCrossEntropyLossLayer {\n    input: \"labelProbability\"\n    target: \"labelProbability_true\"\n  }\n}\nshuffle {\n  defaultValue: true\n}\n<\/code><\/pre>\n<p>As expected, the model now has a <code>categoricalCrossEntropyLossLayer<\/code>.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> The <code>lossLayers<\/code> structure is actually a <em>list<\/em> of layers. This means a Core ML neural network can have more than one loss layer, for example to combine a cross-entropy loss for classification with an MSE loss for regression (perhaps of bounding box coordinates or something similar). However, there is currently no way to weight the different loss terms so that the classification loss counts heavier than the regression loss or vice versa.<\/p>\n<\/div>\n<p>A loss layer usually has two inputs:<\/p>\n<ol>\n<li>the predicted value, given here by <code>input<\/code>, and<\/li>\n<li>the true value, also known as the <code>target<\/code>.<\/li>\n<\/ol>\n<p>The input to this layer is the predicted probabilities from the softmax. The target will receive the training example\u2019s true label.<\/p>\n<p>You may wonder why the target is called <code>\"labelProbability_true\"<\/code>, even though the training input for the true label is named <code>\"label\"<\/code>? It certainly confused me\u2026 I had expected that these two things needed to be the same. How else would Core ML know which training input to connect to the loss layer?<\/p>\n<p>It turns out, classifiers are a bit special in Core ML.<\/p>\n<p>Core ML models can have a <code>predictedFeatureName<\/code> property. This is the primary predicted output feature. In the case of a classifier, <code>predictedFeatureName<\/code> is set to the output that predicts the best-scoring class label. In our model, that is the <code>\"label\"<\/code> output.<\/p>\n<p>(There also is a <code>predictedProbabilitiesName<\/code> property that has the name of the output with the probabilities dictionary. This allows you to rename your outputs to whatever you like but Core ML will still know which is which.)<\/p>\n<p>By setting the name of the training input to the value from <code>predictedFeatureName<\/code>, which in our case is <code>\"label\"<\/code>, Core ML lets you pass in the true label <strong>as a string<\/strong>. That\u2019s actually quite convenient, as the cross-entropy loss normally doesn\u2019t accept string values! Core ML will automatically map the string to the correct categorical index and then pass it into the target of the cross-entropy loss layer.<\/p>\n<p>Mystery solved: for a classifier, the name the loss layer\u2019s target is set to isn\u2019t actually important. Core ML sees that the training input is a string and has the name from <code>predictedFeatureName<\/code>, and so it will internally connect it to the cross-entropy layer\u2019s target.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> If your labels aren\u2019t strings, you\u2019ll need to pass in the true label using an <code>MLMultiArray<\/code> that contains just a single element. This element is the categorical index of the label (there\u2019s no need to one-hot encode the labels). The data type of the <code>MLMultiArray<\/code> can be <code>.int32<\/code>, <code>.float32<\/code>, or <code>.double<\/code>. In this case, the name of the loss layer\u2019s target <em>is<\/em> important and must match the name of a training input. If you\u2019re not training a classifier, i.e. if you\u2019re using the MSE loss, you\u2019ll also have to pass in the target using an <code>MLMultiArray<\/code>.<\/p>\n<\/div>\n<p>Back to <strong>make_nn.py<\/strong>. Even though the training inputs were added automatically, they don\u2019t have a human-readable description yet, so give them one:<\/p>\n<pre><code class=\"language-python\">builder.spec.description.trainingInput[0].shortDescription = \"Example image\"\nbuilder.spec.description.trainingInput[1].shortDescription = \"True label\"\n<\/code><\/pre>\n<h3>The optimizer and its hyperparameters<\/h3>\n<p>As is usual in supervised learning, after computing the loss for a mini-batch of training examples, Core ML will backpropagate through the neural network and use an optimizer to adjust the model\u2019s learned parameters.<\/p>\n<p>Core ML supports two optimizers:<\/p>\n<ul>\n<li>SGD (Stochastic Gradient Descent)<\/li>\n<li>Adam<\/li>\n<\/ul>\n<p>To add an SGD optimizer to the model, you\u2019d do the following:<\/p>\n<pre><code class=\"language-python\">from coremltools.models.neural_network import SgdParams\nsgd_params = SgdParams(lr=0.001, batch=8, momentum=0)\nsgd_params.set_batch(8, [1, 2, 8, 16])\nbuilder.set_sgd_optimizer(sgd_params)\n<\/code><\/pre>\n<p>Creating an Adam optimizer is very similar:<\/p>\n<pre><code class=\"language-python\">from coremltools.models.neural_network import AdamParams\nadam_params = AdamParams(lr=0.001, batch=8, beta1=0.9, beta2=0.999, eps=1e-8)\nadam_params.set_batch(8, [1, 2, 8, 16])\nbuilder.set_adam_optimizer(adam_params)\n<\/code><\/pre>\n<p>The main difference is that Adam has more options. <strong>make_nn.py<\/strong> uses SGD but I\u2019ve included the Adam code here in case you want to experiment with which optimizer works best.<\/p>\n<p>The values you pass into the <code>SgdParams<\/code> or <code>AdamParams<\/code> constructor are the default settings for the different optimizer hyperparameters. Here, the learning rate or <code>lr<\/code> is 0.001 by default and the mini-batch size is 8.<\/p>\n<p>Inside the app you can override these default values, but any new value you choose must meet the constraints defined in the mlmodel file.<\/p>\n<p>For the learning rate, momentum, beta1, beta2, and epsilon, the range of allowed values is between 0 and 1. The mini-batch size by default only allows the value you passed into <code>SgdParams<\/code> or <code>AdamParams<\/code>. To change these constraints, you have to call <code>set_batch()<\/code> or <code>set_lr()<\/code>, etc. Here, we\u2019re telling Core ML that we will allow mini-batches of size 1, 2, 8, and 16.<\/p>\n<p>There are other hyperparameters that you need to set as well, most notably the number of epochs:<\/p>\n<pre><code class=\"language-python\">builder.set_epochs(10, [1, 10, 50])\n<\/code><\/pre>\n<p>When you tell Core ML to train the neural network, it always runs for the number of epochs that is defined in the mlmodel file. Here, we set the number of epochs to be 10 by default, but we\u2019ll allow the app to change this to 1 epoch or 50 epochs too.<\/p>\n<p>Another hyperparameter you can set is <code>shuffle<\/code>. By default this is true, which means Core ML will automatically randomly reorder the training examples each epoch. You can also set the random <code>seed<\/code> used for shuffling.<\/p>\n<p>All right, now the mlmodel is updatable and can be trained by Core ML on the device! Finally, save the model to a new mlmodel file:<\/p>\n<pre><code class=\"language-python\">coremltools.utils.save_spec(builder.spec, \"HandsTuri.mlmodel\")\n<\/code><\/pre>\n<h3>Brainwashing the classifier<\/h3>\n<p>OK, that was step one: we made the Turi Create version of the model updatable.<\/p>\n<p>That\u2019s usually what you\u2019d do for your own apps: take an existing model that already works well on generic data, and make it updatable so that it can be trained on the user\u2019s own data.<\/p>\n<p>But for the purposes of this demo app, we also want to have a version of the model whose classifier layer is not trained yet at all. That way, we can see how hard it is to train a classifier layer from scratch on the device.<\/p>\n<p>To do this, we\u2019ll take the Turi model, leave the SqueezeNet part of the network as-is, but overwrite the learned parameters \u2014 also known as the weights and biases \u2014 of the fully-connected layer.<\/p>\n<p>Before training, layers are usually initialized with small random values. To create an untrained version of the fully-connected layer, we\u2019ll simply replace its learned parameters with random values.<\/p>\n<p>We start out from the HandsTuri model because it has all the stuff we need. (The following code is also in <strong>make_nn.py<\/strong>.)<\/p>\n<pre><code class=\"language-python\">model = coremltools.models.MLModel(\"HandsTuri.mlmodel\")\nmodel.short_description = \"\"\nspec = model._spec\nlayer = spec.neuralNetworkClassifier.layers[-2]\n<\/code><\/pre>\n<p>The fully-connected layer is the second-to-last one, so we can get it with <code>layers[-2]<\/code>. The last layer, <code>layers[-1]<\/code>, is the softmax layer. We don\u2019t need to change that because softmax has no learnable parameters.<\/p>\n<p>To replace the learned parameters of the fully-connected layer, do the following:<\/p>\n<pre><code class=\"language-python\">import numpy as np\n\nW_old = np.array(layer.innerProduct.weights.floatValue)\nB_old = np.array(layer.innerProduct.bias.floatValue)\n\nfan_in = layer.innerProduct.inputChannels\nbound = np.sqrt(6.0 \/ fan_in)\nW_new = np.random.uniform(-bound, bound, W_old.shape)\n\nbound = 1 \/ np.sqrt(fan_in)\nB_new = np.random.uniform(-bound, bound, B_old.shape)\n\nlayer.innerProduct.weights.ClearField(\"floatValue\")  \nlayer.innerProduct.weights.floatValue.extend(W_new)\n\nlayer.innerProduct.bias.ClearField(\"floatValue\")  \nlayer.innerProduct.bias.floatValue.extend(B_new)\n\ncoremltools.utils.save_spec(spec, \"HandsEmpty.mlmodel\")\n<\/code><\/pre>\n<p>The type of initialization we\u2019re using here is so-called \u201cKaiming\u201d initialization. It\u2019s typical for fully-connected layers (although any kind of small random numbers would have worked here).<\/p>\n<p>Thanks to this random initialization, if you try out the untrained model in the app\u2019s Evaluate screen, it will only get around 10% accuracy \u2014 it is just randomly guessing between the ten possible classes.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> The exact accuracy from the untrained model can vary as it largely depends on the random numbers picked by <strong>make_nn.py<\/strong>. Every time you run <strong>make_nn.py<\/strong>, the random initialization will be slightly different, and so will the predictions made by HandsEmpty.mlmodel. Sometimes you may get less than 10% accuracy, sometimes more. The more test images you have, the more likely it is that you\u2019ll end up at exactly 10%.<\/p>\n<\/div>\n<p>And that\u2019s it, now we have two new mlmodel files, one that was trained for a bit already with Turi Create, and one whose classifier is still completely untrained.<\/p>\n<p>If you open HandsEmpty.mlmodel or HandsTuri.mlmodel in Xcode, it looks like this:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-training\/XcodeNeuralNetwork@2x.png\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-training\/XcodeNeuralNetwork.png\" alt=\"The neural network in the Xcode mlmodel viewer\"\/><\/a><br \/>\n<\/figure>\n<p>The <strong>Update<\/strong> section shows the training inputs for the model.<\/p>\n<p>The <strong>Parameters<\/strong> section lists all the hyperparameters. While for the k-NN model this had just one parameter \u2014 the number of neighbors \u2014 for a neural network it shows a lot more stuff.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> Just as with the k-NN version, the demo app won\u2019t make changes directly to HandsEmpty or HandsTuri.mlmodel. The first time you run the app, it copies HandsEmpty.mlmodelc from the app bundle into the Documents directory as <strong>NeuralNetwork.mlmodelc<\/strong>. The app also does this if you tap <strong>Reset to Empty Model<\/strong>, allowing you to restart training on a fresh copy of the model. It is NeuralNetwork.mlmodelc that will actually be trained. To continue training the model from Turi Create, tap <strong>Reset to Turi Model<\/strong> to copy HandsTuri.mlmodelc from the app bundle on top of NeuralNetwork.mlmodelc instead.<\/p>\n<\/div>\n<h2>Back to the app<\/h2>\n<p>Over the course of this series we\u2019ve been building a simple demo app that lets you collect training data and test data, and lets you evaluate the different models on the test set. Last time we added training for the k-NN model, this time we\u2019ll add the training code for the neural network.<\/p>\n<p>If you haven\u2019t yet, <a href=\"https:\/\/github.com\/hollance\/coreml-training\">grab the code from the GitHub repo<\/a> and follow along. (The app runs fine on the iOS Simulator in case you don\u2019t have a compatible device.)<\/p>\n<p>The <strong>Training<\/strong> screen for the neural network looks like this:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-training\/TrainNNScreen@2x.jpg\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-training\/TrainNNScreen.jpg\" alt=\"The Train Neural Network screen\"\/><\/a><br \/>\n<\/figure>\n<p>There are buttons to start and stop training, a slider that lets you set the learning rate, a graph of the training loss, and a table with statistics for every epoch.<\/p>\n<p>The training loss (<strong>tr.loss<\/strong> in the table) is what Core ML reports back to us after each epoch. This is also what is plotted in the graph.<\/p>\n<p>The validation loss (<strong>val.loss<\/strong>) and accuracy (<strong>val.acc<\/strong>) are computed on the test set. This uses the same <code>Predictor<\/code> class that\u2019s also used by the Evaluate screen, <a href=\"https:\/\/machinethink.net\/blog\/coreml-training-part2\/\">see part 2 for more info<\/a>.<\/p>\n<p>The screenshot above shows a model that has been trained for a number of epochs. If you haven\u2019t trained yet (or after you press <strong>Reset to Empty Model<\/strong>), the screen is mostly empty.<\/p>\n<p>The UI of this screen is implemented by <strong>TrainNeuralNetworkViewController.swift<\/strong>. As usual, the view controller code isn\u2019t really interesting. All the good stuff happens in <strong>NeuralNetworkTrainer.swift<\/strong> plus a handful of helper classes.<\/p>\n<p>Try it out! Run the app, press <strong>Train<\/strong> to go to the Train Neural Network screen. Tap <strong>10 Epochs<\/strong> to start training and see what happens. :\u2013)<\/p>\n<div class=\"note\">\n<p><strong>Tip:<\/strong> Recall that when you run the app for the first time, it copies the mlmodel files from the app bundle into the app\u2019s Documents folder. If you made any changes to <strong>make_nn.py<\/strong> or the mlmodel files, first press <strong>Reset to Empty Model<\/strong> in the app\u2019s home screen. This will copy the latest version of the model from the app bundle again. Without this, the app will keep using the old version.<\/p>\n<\/div>\n<p>After training completes, the previous version of the model is overwritten by the updated one. The training history is saved to a file as well, so you can come back to this screen to resume training later. (For reasons I\u2019ll go into, if you press Stop to abort training, the updated model is discarded and you actually lose all the work.)<\/p>\n<p>Now let\u2019s see how the training actually works!<\/p>\n<h3>Loading the training data<\/h3>\n<p>In <a href=\"https:\/\/machinethink.net\/blog\/coreml-training-part3\/\">part 3<\/a>, you\u2019ve seen that training a k-NN model was done by creating an <code>MLUpdateTask<\/code> and giving it an <code>MLBatchProvider<\/code> with the training data. The whole thing took only a handful lines of code. Training a neural network happens in much the same way, except the process is slightly more involved.<\/p>\n<p>The main difference is that k-NN simply memorizes the training examples, so you need to show it each example only once. But neural networks use an <strong>iterative learning process<\/strong> \u2014 a \u201ctraining loop\u201d \u2014 and so they need to see each example many times over.<\/p>\n<p>Let\u2019s begin with the <code>MLBatchProvider<\/code>. With k-NN you typically train on one or a few examples at a time. It\u2019s easy enough to load those examples into memory and put them into an <code>MLArrayBatchProvider<\/code> object. However, for a neural network you may have more training examples than can fit into memory at a time. There\u2019s no way you could fit them all into an <code>MLArrayBatchProvider<\/code>.<\/p>\n<p>The solution is to write your own <code>MLBatchProvider<\/code> class. Even if the amount of training data is limited \u2014 as in the demo app \u2014 another good reason for creating your own batch provider is that this lets you do on-the-fly data augmentation.<\/p>\n<p>The code for our own batch provider is in <strong>TrainingBatchProvider.swift<\/strong>. It\u2019s simpler than you may think, here\u2019s the entire thing:<\/p>\n<pre><code class=\"language-swift\">class TrainingBatchProvider: MLBatchProvider {\n  let imageLoader: ImageLoader\n\n  init(imageLoader: ImageLoader) {\n    self.imageLoader = imageLoader\n  }\n\n  var count: Int {\n    imageLoader.dataset.count\n  }\n\n  func features(at index: Int) -&gt; MLFeatureProvider {\n    guard let featureValue = try? imageLoader.featureValue(at: index) else {\n      print(\"Could not load image at index (index)\")\n      return failure()\n    }\n\n    let trueLabel = imageLoader.dataset.label(at: index)\n\n    guard let pixelBuffer = featureValue.imageBufferValue else {\n      print(\"Could not get pixel buffer for image at (index)\")\n      return failure()\n    }\n\n    return HandsEmptyTrainingInput(image: pixelBuffer, label: trueLabel)\n  }\n\n  private func failure() -&gt; MLFeatureProvider {\n    return try! MLDictionaryFeatureProvider(dictionary: [:])\n  }\n}\n<\/code><\/pre>\n<p>All you have to do is implement the <code>count<\/code> property and the <code>features(at:)<\/code> method.<\/p>\n<p>Notice that, even though this is called a <em>batch<\/em> provider, it only returns a single training example at a time. Core ML will repeatedly call <code>features(at:)<\/code> with different indices to fill up the mini-batch. If shuffling is enabled, those indices will be picked randomly.<\/p>\n<p>The <code>MLBatchProvider<\/code> actually doesn\u2019t know anything about batches (!) or about how the images get stored inside a batch. So the name is a little misleading: it doesn\u2019t provide batches, only things that go inside batches. (It\u2019s also not notified when a new batch or epoch starts.)<\/p>\n<p>Since we already have an <code>ImageLoader<\/code> class, our batch provider simply uses that to load the images. <code>ImageLoader<\/code> also performs the data augmentation if that is enabled (more about this later).<\/p>\n<p>We also provide the true labels for each example.<\/p>\n<p>Each training example consists of an image and its label. Core ML expects you to return both in a single <code>MLFeatureProvider<\/code> object. Rather than constructing the <code>MLFeatureProvider<\/code> objects from scratch, we simply use the Xcode-generated class <code>HandsEmptyTrainingInput<\/code> (which already implements <code>MLFeatureProvider<\/code>).<\/p>\n<p>You could also have written the following:<\/p>\n<pre><code class=\"language-swift\">let inputs: [String: Any] = [ \"image\": featureValue, \n                              \"label\": trueLabel ]\nreturn try! MLDictionaryFeatureProvider(dictionary: inputs)\n<\/code><\/pre>\n<p>Personally, I find using the generated class a bit more convenient but either works.<\/p>\n<p>There\u2019s no way to return an error but there\u2019s always the possibility that loading an image \u2014 or whatever data you have \u2014 fails. It turns out that returning an empty <code>MLFeatureProvider<\/code> object, like we do here using the <code>failure()<\/code> function, will cause training to immediately end with a <code>.failed<\/code> state. So that\u2019s one way you can handle data loading errors.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> Core ML will automatically take care of shuffling the training data if that option is enabled. My <code>ImageLoader<\/code> class has its own option to shuffle the data. When I first wrote this code during the iOS 13 beta period, the shuffle hyperparameter didn\u2019t exist yet. I left it in the code, so you can see how to implement shuffling yourself if you ever need to.<\/p>\n<p>The <code>ImageLoader<\/code> also has a way to split the data into batches, but we\u2019re not using that during training. Instead, we let Core ML drive the batching process. So my <code>ImageLoader<\/code> class is actually more complicated than it needs to be. \ud83d\ude2d<\/p>\n<\/div>\n<h3>Creating the update task<\/h3>\n<p>Now that we have our training data and can load it into mini-batches, we can start the training process. This is done by making an <code>MLUpdateTask<\/code>. It\u2019s the same class you used for training the k-NN model, but this time you\u2019ll be using a different initializer.<\/p>\n<p>The code looks something like this:<\/p>\n<pre><code class=\"language-swift\">let trainingData = TrainingBatchProvider(imageLoader: trainingLoader)\n\nlet config = MLModelConfiguration()\nconfig.computeUnits = .all\n\nlet progressHandler = { (context: MLUpdateContext) in\n  \/* TODO *\/\n}\n\nlet completionHandler = { (context: MLUpdateContext) in\n  \/* TODO *\/\n}\n\nlet handlers = MLUpdateProgressHandlers(\n                    forEvents: [.trainingBegin, .miniBatchEnd, .epochEnd],\n                    progressHandler: progressHandler,\n                    completionHandler: completionHandler)\n\nupdateTask = try MLUpdateTask(forModelAt: modelURL,\n                              trainingData: trainingData,\n                              configuration: config,\n                              progressHandlers: handlers)\nupdateTask?.resume()\n<\/code><\/pre>\n<p>You can find the full version of this code in <strong>NeuralNetworkTrainer.swift<\/strong>, in the <code>train<\/code> function.<\/p>\n<p>The k-NN version of <code>MLUpdateTask<\/code> took a completion handler that was invoked almost immediately because training k-NN is very fast. But as we discussed, neural networks are trained in a loop over many epochs. That\u2019s why you need to provide two closures here: a handler for progress updates as well as the usual completion handler.<\/p>\n<p>Both handlers will receive an <code>MLUpdateContext<\/code> object that describes the current state of the training process. One of the useful things you can get from this context object is the current loss value.<\/p>\n<p>You can register the progress handler for the following events:<\/p>\n<ul>\n<li><code>.trainingBegin<\/code>: happens once when training starts<\/li>\n<li><code>.miniBatchEnd<\/code>: happens after every mini-batch<\/li>\n<li><code>.epochEnd<\/code>: happens after every epoch<\/li>\n<\/ul>\n<p>(There are no \u201cepochStart\u201d or \u201cminiBatchStart\u201d callbacks like in Keras.)<\/p>\n<p>Besides these progress and completion handlers, the <code>MLUpdateTask<\/code> also needs:<\/p>\n<ul>\n<li>the URL to the <strong>mlmodelc<\/strong> file that will be trained,<\/li>\n<li>the <code>MLBatchProvider<\/code> object that will load the training data, and<\/li>\n<li>an <code>MLModelConfiguration<\/code> object.<\/li>\n<\/ul>\n<p>The configuration object has a <code>computeUnits<\/code> property that tells Core ML whether it\u2019s allowed to use the GPU or the Neural Engine (although ultimately it\u2019s up to Core ML to choose which processor it uses). You can also use the configuration object to change the hyperparameters (see below).<\/p>\n<p>Once the <code>MLUpdateTask<\/code> is constructed, call <code>resume()<\/code> to start training.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> You don\u2019t have to load the Core ML model into memory first before you can train it \u2014 i.e. there\u2019s no need to have an active <code>MLModel<\/code> object. Training always works from the <strong>mlmodelc<\/strong> files that are stored somewhere in your app\u2019s local directory.<\/p>\n<\/div>\n<h3>The training flow chart<\/h3>\n<p>This is the flow of what happens after you call <code>updateTask.resume()<\/code>:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-training\/Flowchart@2x.png\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-training\/Flowchart.png\" alt=\"The training process\"\/><\/a><br \/>\n<\/figure>\n<p><code>MLUpdateTask.resume()<\/code> immediately returns and the training loop happens in a background thread. You don\u2019t need to put this on a background queue yourself. The progress and completion handlers are also called on a background thread.<\/p>\n<p>It\u2019s important to realize that calling <code>MLUpdateTask.resume()<\/code> will always train for the number of epochs that is defined in the mlmodel file. In this demo app, that is 10 epochs.<\/p>\n<p>The mini-batch size and whether the data is shuffled or not, is also defined in the mlmodel file. For our model, <code>miniBatchSize<\/code> is 8 and <code>shuffle<\/code> is true.<\/p>\n<p>Because there are 30 images in the training set (unless you added your own), each epoch consists of 4 batches. With 30 images, you\u2019d expect the last batch to have only 6 images instead of 8, but Core ML fills up the remainder of the batch by starting from the beginning again. This is perfectly fine, but it means the model might see some images more than once in the epoch.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> In the first mini-batch of the first epoch, the first image is requested twice for some reason. I\u2019m assuming Core ML does this on purpose to inspect what the data will be like. It\u2019s not a big deal, but the result from the very first call to your <code>MLBatchProvider<\/code> might not actually be used. Also, the batch provider\u2019s <code>.count<\/code> is called a whole bunch of times, so make sure this method isn\u2019t doing heavy work.<\/p>\n<\/div>\n<p>With shuffling enabled, the order of the shuffled images is always the same. That\u2019s because the random seed is also hardcoded in the mlmodel file. In <strong>make_nn.py<\/strong> you could add the following line to change the seed:<\/p>\n<pre><code class=\"language-python\">builder.set_shuffle(seed=1234)\n<\/code><\/pre>\n<p>By default the seed is <code>None<\/code>. Usually a seed of <code>None<\/code> means you\u2019re not setting the seed explicitly (for extra randomness), but I guess in Core ML if you don\u2019t set the seed it chooses some fixed number anyway (I checked and it\u2019s not 0).<\/p>\n<p>With a fixed random seed, repeating the training several times will give the exact same results each time. You can override the seed using the model configuration parameters, just like any of the other hyperparameters. For true randomness, you can set the seed to the current time, for example.<\/p>\n<h3>The progress handler<\/h3>\n<p>In the demo app, the <code>MLUpdateTask<\/code>\u2019s progress handler looks something like this:<\/p>\n<pre><code class=\"language-swift\">let progressHandler = { (context: MLUpdateContext) in\n  switch context.event {\n  case .trainingBegin:\n    print(\"Training begin\")\n\n  case .miniBatchEnd:\n    let batchIndex = context.metrics[.miniBatchIndex] as! Int\n    let batchLoss = context.metrics[.lossValue] as! Double\n    print(\"Mini batch (batchIndex), loss: (batchLoss)\")\n\n  case .epochEnd:\n    let epochIndex = context.metrics[.epochIndex] as! Int\n    let trainLoss = context.metrics[.lossValue] as! Double\n\n    let predictor = Predictor(model: context.model)\n    let (valLoss, valAcc) = predictor.evaluate(loader: self.validationLoader)\n\n    \/\/ Tell the view controller about the results for this epoch:\n    callback(.epochEnd(trainLoss: trainLoss, validationLoss: valLoss, \n                       validationAccuracy: valAcc))\n  default:\n    print(\"Unknown event\")\n  }\n}\n<\/code><\/pre>\n<p>Because the progress handler can be called for different reasons, first we look at <code>context.event<\/code> to see what\u2019s happening.<\/p>\n<p>The <code>.trainingBegin<\/code> event isn\u2019t really that interesting. You could use this to reset some counters or other properties.<\/p>\n<p>The <code>.miniBatchEnd<\/code> event is useful for when you want to do something after each mini-batch. For this kind of event, the <code>context.metrics<\/code> dictionary contains two items: <code>.miniBatchIndex<\/code> tells you which mini-batch this is for the current epoch, and <code>.lossValue<\/code> gives the training loss for this mini-batch.<\/p>\n<p>But other than that, there isn\u2019t much you can do. You don\u2019t get access to the actual mini-batch\u2019s training data, so you can\u2019t compute other metrics such as the training accuracy over the mini-batch. Not a major issue for on-device training, but good to be aware of these limitations.<\/p>\n<p>The most useful event is <code>.epochEnd<\/code>. This is called after each run through the entire training set. This is a good time to compute some metrics across the test set. In the demo app, we\u2019re using the <code>Predictor<\/code> class for this.<\/p>\n<p>You may recall that <code>Predictor<\/code> is used by the Evaluate screen to run the model on the test set images. It needs an <code>MLModel<\/code> object for this \u2014 obviously, not just <em>any<\/em> <code>MLModel<\/code> but the one that\u2019s currently being trained. You can get the model-under-training from the context, using <code>context.model<\/code>.<\/p>\n<p>It\u2019s important that you use the <code>MLModel<\/code> instance from the context, as that has the most recent training changes. Any <code>MLModel<\/code> instances that you created in your app before training started do not automatically get these changes!<\/p>\n<p>(Also note that at this point the updated model hasn\u2019t been saved to a file yet, so the existing mlmodelc is still the old version.)<\/p>\n<p>After each epoch, the <code>TrainNeuralNetworkViewController<\/code> needs to display the training loss, validation loss, and validation accuracy in its table view, so we notify it through a closure I named the <code>callback<\/code>. This is just UI stuff and is unrelated to Core ML\u2019s training APIs.<\/p>\n<h3>Computing the validation loss<\/h3>\n<p>For the purpose of computing the loss over the validation set, I added a new <code>evaluate()<\/code> method to <code>Predictor<\/code>. See <strong>Predictor.swift<\/strong> for the full details. It just loops through the test set and makes predictions in batches. (These are batches from <code>ImageLoader<\/code>, not from the <code>MLBatchProvider<\/code> we\u2019re using for training.)<\/p>\n<p>The interesting code I want to highlight is the computation of the loss value:<\/p>\n<pre><code class=\"language-swift\">var runningLoss: Double = 0\n\nwhile let batch = try? loader.nextBatch() {\n  if let batchPredictions = predict(batch: batch) {\n    ...\n    let probabilityForTrueClass = batchPredictions[i].probabilities[trueLabel]!\n    let crossEntropy = -log(probabilityForTrueClass.doubleValue + 1e-100)\n    runningLoss += crossEntropy\n  }\n}\n\nlet validationLoss = runningLoss \/ Double(exampleCount)\n<\/code><\/pre>\n<p>Core ML automatically computes the loss over the training set for us, but to get the loss over the validation set we have to do the math ourselves. Fortunately, it\u2019s fairly straightforward to compute the cross-entropy loss given the dictionary of predicted probabilities.<\/p>\n<p>We\u2019re only interested in the probability that the model predicts for the true label. The cross-entropy loss for a single example is the logarithm of that probability. (We add a small term to the probability to prevent <code>log(0)<\/code>, which is infinity.)<\/p>\n<p>The total cross-entropy loss over the entire dataset is simply the average of the losses for the individual examples.<\/p>\n<p>I showed this to point out that, just because Core ML doesn\u2019t give you an API for something, this doesn\u2019t necessarily mean you can\u2019t implement it by yourself. \ud83e\udd13<\/p>\n<h3>The completion handler<\/h3>\n<p>The <code>MLUpdateTask<\/code> completion handler is called just once, after the last epoch completes or if training is aborted due to an error. Not sure why they didn\u2019t add a <code>.trainingEnd<\/code> event for this and made it a separate handler, but \/shrug.<\/p>\n<p>The demo app\u2019s completion handler looks like this:<\/p>\n<pre><code class=\"language-swift\">let completionHandler = { (context: MLUpdateContext) in\n  if context.task.state == .failed {\n    callback(.error)\n    return\n  }\n\n  let trainLoss = context.metrics[.lossValue] as! Double\n  print(\"Final loss: (trainLoss)\")\n\n  self.saveUpdatedModel(context.model, to: self.modelURL)\n\n  \/\/ Tell the view controller we're done; also pass the new MLModel instance.\n  callback(.completed(updatedModel: context.model))\n}\n<\/code><\/pre>\n<p>If something went wrong, the completion handler is still called but the <code>state<\/code> of the <code>MLUpdateTask<\/code> is set to <code>.failed<\/code>. You may end up in this <code>.failed<\/code> state if the batch provider returns an invalid <code>MLFeatureProvider<\/code> object, for example. If this happens we\u2019ll tell the UI that there was an error (through the <code>callback<\/code> closure).<\/p>\n<p>Assuming all went well, the completion handler is a good place to save the updated model to a file. We\u2019re using the following helper function:<\/p>\n<pre><code class=\"language-swift\">private func saveUpdatedModel(_ model: MLModel &amp; MLWritable, to url: URL) {\n  do {\n    let tempURL = urlForModelInDocumentsDirectory(\"tempNeuralNetwork\")\n    try model.write(to: tempURL)\n    _ = try FileManager.default.replaceItemAt(url, withItemAt: tempURL)\n  } catch {\n    print(\"Error saving neural network model to (url):\", error)\n  }\n}\n<\/code><\/pre>\n<p>I wanted to show this because of the <code>MLModel &amp; MLWritable<\/code>. The model object from <code>context.model<\/code> also conforms to the <code>MLWritable<\/code> protocol, which means you can call <code>model.write(to:)<\/code> on it. A regular <code>MLModel<\/code> is not <code>MLWritable<\/code>, so this only works from within an update context.<\/p>\n<p>In the demo app we\u2019re simply overwriting the existing mlmodelc files, but there\u2019s nothing stopping you from saving several different checkpoints.<\/p>\n<div class=\"note\">\n<p><strong>Tip:<\/strong> If you\u2019re training for many epochs, the <code>.epochEnd<\/code> event is also a good point to save a model checkpoint file (after every 10 epochs, for example). We\u2019re not doing that in the demo app, but feel free to add this logic yourself.<\/p>\n<\/div>\n<p>The final thing that happens in the completion handler is a callback to let the <code>TrainNeuralNetworkViewController<\/code> know that training has completed. Note that we pass along the <code>MLModel<\/code> instance from <code>context.model<\/code>. The view controller has its own <code>MLModel<\/code> instance but that is the previous version that hasn\u2019t been updated yet. You could tell the view controller to instantiate the <code>MLModel<\/code> again but it\u2019s easier (and more efficient) to grab the latest version from the update context.<\/p>\n<h3>But which training loss?<\/h3>\n<p>As you\u2019ve seen above, you can get the training loss from the context object using <code>context.metrics[.lossValue] as! Double<\/code>. The docs say this is the \u201ccurrent loss\u201d, but what exactly does that mean?<\/p>\n<p>It looks like the reported loss at the end of an epoch is just the loss from the <strong>most recent batch<\/strong>, not the average loss over the batches. Likewise, the reported loss at the end of training is just the loss from the last batch in the last epoch.<\/p>\n<p>In theory, if you give <code>Predictor<\/code> in <strong>NeuralNetworkTrainer.swift<\/strong> the training set instead of the validation set, the \u201cvalidation loss\u201d that is computed after each epoch should be similar to the training loss reported by Core ML. After all, they\u2019re both computed using the same formula on the same set of images (although not exactly at the same time).<\/p>\n<p>As it turns out, the two losses are not the same\u2026 On the first few epochs this makes some sense: you\u2019d expect there to be a difference as training is really unstable in the beginning. But over time the two losses should converge.<\/p>\n<p>Taking an average over the mini-batch losses seems to be closer to the loss computed by hand, so from this I\u2019m concluding that Core ML doesn\u2019t keep a running loss but just reports whatever the loss was on the very latest mini-batch it processed.<\/p>\n<p>Anyway, it\u2019s probably not very important to know this but I was curious.<\/p>\n<p>The loss is the only metric we get. There\u2019s no way to compute the classification accuracy of a mini-batch, at least not one without resorting to hacks. I guess this is fine \u2014 we mainly just care about the training loss going down as indicator that the model is actually learning something, and we can compute the validation metrics ourselves to make sure it\u2019s learning the right thing.<\/p>\n<h2>Changing the hyperparameters<\/h2>\n<p>Training a neural network involves plenty of hyperparameters that you, as the model designer, need to choose in advance.<\/p>\n<p>The models used in this blog post have the following hyperparameters:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-training\/Hyperparameters@2x.png\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-training\/Hyperparameters.png\" alt=\"The hyperparameters for the neural network model\"\/><\/a><br \/>\n<\/figure>\n<p>(If you changed <strong>make_nn.py<\/strong> to use Adam instead of SGD, you\u2019ll see different hyperparameters here.)<\/p>\n<p>The values for these hyperparameters are set in the mlmodel file. This is useful because Core ML can simply use these settings when you tell it to train. The app developer won\u2019t need to worry about choosing good values for these parameters.<\/p>\n<p>Invoking the <code>MLUpdateTask<\/code> will train for the number of epochs specified, using mini-batches of the given size, using the specified learning rate.<\/p>\n<p>However, you may want a bit more control over the training process. That\u2019s what the <code>MLModelConfiguration<\/code> object is for. You can use this class to choose between running the model on the CPU \/ GPU \/ Neural Engine, but it also has a <code>parameters<\/code> dictionary for overriding the hyperparameters.<\/p>\n<p>This is how you use it:<\/p>\n<pre><code>let parameters: [MLParameterKey: Any] = [\n  .miniBatchSize: 2,\n  .epochs: 3,\n  .shuffle: false,\n  .seed: 4321,\n  .learningRate: learningRate,\n]\n\nlet config = MLModelConfiguration()\nconfig.computeUnits = .all\nconfig.parameters = parameters\n<\/code><\/pre>\n<p>And then you create the <code>MLUpdateTask<\/code> with this <code>config<\/code> object.<\/p>\n<p>In the demo app, this is used to let the user set the learning rate (using the slider) and number of epochs.<\/p>\n<p>Note that <code>config.parameters<\/code> is initially nil. To override the hyperparameters from the model, you need to create the parameters dictionary yourself.<\/p>\n<p>(If you already have an <code>MLModel<\/code> instance, you can also use <code>model.configuration<\/code> to get its <code>MLModelConfiguration<\/code> object. But remember that Core ML never trains the <code>MLModel<\/code> instance directly, only its mlmodelc files.)<\/p>\n<p>When you set the hyperparameters, keep in mind that you can\u2019t just choose <em>any<\/em> values here. Remember that in <strong>make_nn.py<\/strong> we wrote,<\/p>\n<pre><code>sgd_params.set_batch(8, [1, 2, 8, 16])\n<\/code><\/pre>\n<p>This sets the default value for the <code>.miniBatchSize<\/code> parameter to 8, and we tell Core ML that batch sizes of 1, 2 or 16 are also allowed. If you choose any other number in the app for <code>.miniBatchSize<\/code>, Core ML will throw an exception when you start training.<\/p>\n<p>In <strong>make_nn.py<\/strong> we also set the allowed values for the number of epochs to 1, 10 and 50. The learning rate by default is allowed to be between 0.0 and 1.0, but you can change this with <code>sgd_params.set_lr()<\/code>. Likewise for the other optimizer hyperparameters.<\/p>\n<div class=\"note\">\n<p><strong>Note:<\/strong> If you change any of these values in <strong>make_nn.py<\/strong> and run the script again, don\u2019t forget to tap on <strong>Reset to Empty Model<\/strong> in the app. Otherwise it still keeps using the previous version of the model.<\/p>\n<\/div>\n<h3>Inspecting the hyperparameters<\/h3>\n<p>The Xcode mlmodel viewer only shows the default values of the hyperparameters, but not their allowed values. Fortunately, there is an API that lets you inspect this at runtime, but it\u2019s a little weird.<\/p>\n<p>Recall from part 3 that <code>modelDescription.trainingInputDescriptionsByName<\/code> lets you see what the names and datatypes of the training inputs are. These are also shown in the Xcode mlmodel viewer.<\/p>\n<p>I also mentioned that <code>model.modelDescription.parameterDescriptionsByKey<\/code>, which should give information about the hyperparameters, always returns an empty dictionary. It turns out that <code>parameterDescriptionsByKey<\/code> only works from inside the <code>MLUpdateTask<\/code> progress handler. \ud83e\udd2f<\/p>\n<p>Try it out, add <code>print(model.modelDescription.parameterDescriptionsByKey)<\/code> inside the progress handler and now it will print out the following:<\/p>\n<pre><code class=\"language-nohighlight\">[shuffle: \nparameterKey: shuffle\ndefaultValue: 1, miniBatchSize: \nparameterKey: miniBatchSize\ndefaultValue: 8\nnumericConstraint: \nminValue: 1\nmaxValue: 16\nenumeratedNumbers: {(\n    2,\n    16,\n    8,\n    1\n)}, ...\n<\/code><\/pre>\n<p>This doesn\u2019t tell you what the current values are for these hyperparameters, only what the default and allowed values are. For some strange reason, this information is only filled in when the model is actively being trained.<\/p>\n<p>Another fun thing you can do inside the <code>MLUpdateTask<\/code>\u2019s progress or completion handler is grab the weights and biases of a layer in the model, for example by writing:<\/p>\n<pre><code>do {\n  let multiArray = try context.model.parameterValue(for: \n        MLParameterKey.weights.scoped(to: \"fullyconnected0\")) as! MLMultiArray\n  print(multiArray.shape)\n} catch {\n  print(error)\n}\n<\/code><\/pre>\n<p>You now get an <code>MLMultiArray<\/code> object containing the weights of the layer named <code>\"fullyconnected0\"<\/code>. You can also do this for <code>.biases<\/code>.<\/p>\n<p>This only seems to work for the layers that are trainable. If you use another layer name, for example <code>\"conv10\"<\/code>, this crashes somewhere deep inside Core ML.<\/p>\n<p>These weights and biases appear to be read-only. You can change the contents of the returned <code>MLMultiArray<\/code> but this has no effect on the model or the training procedure (it\u2019s most likely a copy of the actual weights).<\/p>\n<p>Not sure why you\u2019d need this, except maybe for debugging or visualizations, but it\u2019s nice to have anyway.<\/p>\n<h3>Changing the hyperparameters during training<\/h3>\n<p>You\u2019ve seen how to change the values of the hyperparameters <em>before<\/em> training starts, but you can also change some of them during training.<\/p>\n<p>A typical hyperparameter you\u2019d want to change during training is the <strong>learning rate<\/strong>. It\u2019s common to start with a high learning rate and to make it gradually smaller over time \u2014 known as learning rate annealing \u2014 although other schemes are used too (cyclic learning rates, for example).<\/p>\n<p>Inside the progress handler, you can access the <code>MLModel<\/code> object through <code>context.model<\/code>. From the <code>MLModel<\/code> object you can get the <code>MLModelConfiguration<\/code>, which lets you access the hyperparameters. However, the following won\u2019t work:<\/p>\n<pre><code>let newParameters: [MLParameterKey: Any] = [ . . . ]\ncontext.model.configuration.parameters = newParameters\n<\/code><\/pre>\n<p>Core ML simply ignores the new parameters and keeps using the old ones. Instead, what you\u2019re supposed to do is <strong>resume<\/strong> the <code>MLUpdateTask<\/code> from within the progress handler, like so:<\/p>\n<pre><code>let progressHandler = { (context: MLUpdateContext) in\n  switch context.event {\n    \/* previous code omitted *\/\n  }\n\n  \/\/ Simple learning rate annealing:\n  var newParameters = context.parameters\n  let annealed = (newParameters[.learningRate] as! Double) * 0.99\n  newParameters[.learningRate] = annealed\n  context.task.resume(withParameters: newParameters)\n}\n<\/code><\/pre>\n<p>To see the current values of the hyperparameters, don\u2019t look at <code>context.model.configuration.parameters<\/code> \u2014 those are the ones training started out with and will never change. Instead, look at <code>context.parameters<\/code>, this has the active values.<\/p>\n<p>You can\u2019t change all hyperparameters this way, only the <strong>optimizer settings<\/strong>: the learning rate, momentum, etc. Shuffle, mini-batch size, and number of epochs are fixed once training starts.<\/p>\n<h3>Cancelling training<\/h3>\n<p>In the demo app, we cancel training when the user taps the <strong>Stop<\/strong> button, goes back to the main menu, or puts the app into the background.<\/p>\n<p>Given the <code>MLUpdateTask<\/code> instance, you can call <code>cancel()<\/code> on it to interrupt the training process. This immediately stops training. However, be aware that the completion handler does not get called now!<\/p>\n<p>In the progress or completion handler it is possible to look at <code>context.task.state<\/code> to determine whether the task is still running or not. As far as I can tell, inside the progress handler the state always seems to be \u201csuspended\u201d (i.e. paused) and in the completion handler it is either \u201ccompleted\u201d or \u201cfailed\u201d.<\/p>\n<p>I think it would have been useful to get a completion event with the state set to <code>.cancelling<\/code> \u2014 which <em>is<\/em> an existing state flag \u2014 but that\u2019s not the case.<\/p>\n<p>Oh well, at least you can use KVO or <a href=\"https:\/\/machinethink.net\/blog\/coreml-and-combine\/\">Combine<\/a> to keep track of changes to the <code>state<\/code> property.<\/p>\n<p>Because the completion handler isn\u2019t called when cancelling, this also means that any changes to the model will not be saved. So you lose all your work. One way to prevent this is to save a checkpoint after every <em>N<\/em> epochs \u2014 I didn\u2019t bother with this in the demo app but it should be trivial to add.<\/p>\n<p>If you don\u2019t stop the training task when going into the background, Core ML will automatically pause the task and resume when the app becomes active again. I\u2019m not sure I really trust that 100%. I need to experiment more with this to see what happens when the GPU or Neural Engine is used, for example. (On my iPhone XS, Core ML never used the GPU or Neural Engine for training, only the CPU.)<\/p>\n<p>That\u2019s why in the demo app, we cancel the learning task when the app becomes inactive.<\/p>\n<p>Training while in the background is certainly possible, but you\u2019ll need to use the new <a href=\"https:\/\/developer.apple.com\/documentation\/backgroundtasks\">BackgroundTasks framework<\/a> for that (more on this below).<\/p>\n<h2>How well does it work?<\/h2>\n<p>Here\u2019s the graph from one of my training runs:<\/p>\n<figure class=\"image\">\n  <a href=\"https:\/\/machinethink.net\/images\/coreml-training\/Graph@2x.png\"><img decoding=\"async\" src=\"https:\/\/machinethink.net\/images\/coreml-training\/Graph.png\" alt=\"The training loss graph at lr=0.0001\"\/><\/a><br \/>\n<\/figure>\n<p>As you can see, the training loss goes down quite nicely. This means training the model worked!<\/p>\n<p>But the real question is: how good are the predictions from this model on the test set? As you can see from the <strong>val.acc<\/strong> column, in this particular training session it reached 53.33% accuracy. That\u2019s 8 out of 15 correct.<\/p>\n<p>Doesn\u2019t sound too impressive, but it <em>is<\/em> evidence that the model has learned something!<\/p>\n<p>The untrained model only gets 10% correct by random guessing. And even with just 3 classes in the test set, if the model was lazily predicting the same label for all examples, it still wouldn\u2019t score higher than 33%. Therefore, anything over 33% means we\u2019re doing better than guesswork!<\/p>\n<p>To get these results, I played around a bit with different learning rates. To start with a clean slate, I also tapped <strong>Reset to Empty Model<\/strong> before doing each experiment.<\/p>\n<ul>\n<li>\n<p>Using the SGD optimizer, if I put the learning rate at <strong>0.01<\/strong> and train for 20 epochs, the training loss bounces up and down quite a bit but eventually drops to 0. The validation accuracy is 66.67% but the validation loss is really high.<\/p>\n<\/li>\n<li>\n<p>With the learning rate at <strong>0.001<\/strong>, the training loss doesn\u2019t go down to 0 as quickly, but the validation loss is much lower than before. The validation accuracy also seems to be a bit better, occasionally going up to 80% and eventually settling on 73.33%.<\/p>\n<\/li>\n<li>\n<p>With a learning rate of <strong>0.0001<\/strong>, both the training loss and validation loss are much more stable. Naturally, it also takes more epochs to get a good accuracy score. This is the curve from the above picture. Interestingly enough, the validation score now isn\u2019t as good as with the higher learning rates, only 53.33%. But the validation loss <em>is<\/em> lower than before (more about this in a sec).<\/p>\n<\/li>\n<li>\n<p>At an even lower learning rate, <strong>0.00001<\/strong>, the loss curve becomes smoother again, but the validation accuracy never goes over 40%. This learning rate is too low now.<\/p>\n<\/li>\n<\/ul>\n<p>For this model \u2014 and the current dataset \u2014 0.0001 seems to be a good learning rate when using the SGD optimizer. That may seem like a low learning rate, but keep in mind we\u2019re only fine-tuning a single layer.<\/p>\n<div class=\"note\">\n<p><strong>Disclaimer:<\/strong> I must add a warning that you can\u2019t really put too much stock in these numbers because my training set and test set are so small. Does the model really score 80% accuracy or did it just get lucky on this particular handful of images? Answering that question is out of the scope of this blog post series \u2014 we\u2019re just messing around with the new Core ML APIs here, not building a state-of-the-art Rock, Paper, Scissors model. Just know that for real world scenarios you\u2019ll need to use a lot more data for both training and validation to get dependable results!<\/p>\n<\/div>\n<p>Is training the classifier from scratch better than the Turi Create model, which scored 80% on the test set? To be honest, it\u2019s hard to say. It\u2019s possible to get a model that also scores 80% but there is just too little test data to conclude whether that\u2019s a better model or not.<\/p>\n<p>The way Turi Create trains the model is different: it uses L-BFGS to train a logistic regression and then turns that into a neural network afterwards. Our method of training uses gradient descent and backpropagation. It\u2019s quite possible that L-BFGS finds a better solution than SGD.<\/p>\n<p>On the plus side, we have more flexibility. For example, instead of just training the classifier layer, you can also train the convolution layer right before it, <code>\"conv10\"<\/code>. That\u2019s not something Turi Create can do. We can also do more data augmentation.<\/p>\n<p>If you go to the Evaluate screen after training, you can see which test images the model has problems with. It\u2019s mostly the images from the \u270c\ufe0f category, especially the ones that are rotated or flipped. That\u2019s something that should be fixable with data augmentation.<\/p>\n<h3>What\u2019s up with the validation loss?<\/h3>\n<p>You may be wondering why the validation loss on the test set is often so high? The training loss goes to almost zero, while the validation loss stays at something like 3.0 or even higher. That seems odd.<\/p>\n<p>For a classification problem with 10 classes, the expected loss for a classifier that just randomly guesses the answers is <code>-log(1\/10) = 2.3026<\/code>. So how come our validation loss is higher?<\/p>\n<p>This happens because the cross-entropy loss only looks at the predicted probability of the true label. If that probability is high (close to 1), the loss value is close to 0. But the lower this predicted probability, the higher the loss value.<\/p>\n<p>In <code>Predictor.evaluate()<\/code> there is a commented-out print statement. If you enable this, it prints something like the following after training an epoch:<\/p>\n<pre><code class=\"language-nohighlight\">...\n\u270b correct? \u221a, prob: 1.00000, loss: 0.00000\n\u270c\ufe0f correct? \u00d7, prob: 0.00000, loss: 26.52718\n\u270c\ufe0f correct? \u00d7, prob: 0.00000, loss: 17.05890\n\u270c\ufe0f correct? \u221a, prob: 1.00000, loss: -0.00000\n\u270c\ufe0f correct? \u00d7, prob: 0.00000, loss: 26.36917\n\u270c\ufe0f correct? \u221a, prob: 0.79783, loss: 0.22586\n<\/code><\/pre>\n<p>Notice that the predictions that are correct (the ones with the \u221a checkmark) have high probabilities. That\u2019s good. The loss for these is very small.<\/p>\n<p>But on predictions that are not correct (the ones with \u00d7), the predicted probability for the true label is often close to 0 (rounded off to 5 decimals in the output). The model is really wrong about these predictions because we\u2019d expect these predicted probabilities to be close to 1. For these examples, the model predicted a very high probability for some <em>other<\/em> class.<\/p>\n<p>The loss is the average over all the examples. Even though 80% of the predictions are correct in this particular epoch, because the wrong predictions are <em>very<\/em> wrong, the loss still ends up being quite high. You can also see this in the Evaluate screen: the model has very high confidence in the predictions it got wrong.<\/p>\n<p>Here\u2019s an experiment you can do: In <strong>NeuralNetworkTrainer.swift<\/strong>, in the instantiation of <code>validationLoader<\/code>, use the <code>trainingDataset<\/code> instead of <code>validationDataset<\/code>. Now after every epoch the app will compute the accuracy on the training set instead of the test set.<\/p>\n<p>Once the training loss becomes really small (0.001 or smaller), you\u2019ll see that the accuracy on the training set is 100%. <strong>We\u2019re overfitting!<\/strong> Not so strange due to the very small amount of data that we have.<\/p>\n<p>There is no way to add weight decay or other regularization with Core ML (and no dropout layer) so you\u2019ll have to be very careful to either have enough data, or design your model so that overfitting is less of an issue \u2014 smaller models with fewer learned parameters suffer less from overfitting than very large models, for example.<\/p>\n<h3>Try it for yourself!<\/h3>\n<p>I encourage you to run the app and do some experiments yourself.<\/p>\n<p>Try training for a number of epochs, then lower the learning rate using the slider and train some more. What learning rates give good results? Too high and the loss will explode, too low and the model won\u2019t learn anything.<\/p>\n<p>Switch the model to use the Adam optimizer to see what kind of results that gives (you\u2019ll need to modify <strong>make_nn.py<\/strong> for this, as described earlier).<\/p>\n<p>Also try making more layers trainable. The more layers are trained, the better the neural network can adapt to our data. So far, we\u2019ve only trained the very last layer, <code>\"fullyconnected0\"<\/code>. The layer that immediately precedes that is a convolutional layer named <code>\"conv10\"<\/code>. Change <strong>make_nn.py<\/strong> to also make this layer updatable:<\/p>\n<pre><code class=\"language-python\">builder.make_updatable([\"fullyconnected0\", \"conv10\"])\n<\/code><\/pre>\n<div class=\"note\">\n<p><strong>Note:<\/strong> For this particular model, that\u2019s as far as it goes. You can\u2019t train more layers, because of the concatenation layer between <code>\"conv10\"<\/code> and the preceding convolutional layers. Currently, Core ML cannot backpropagate through concat layers.<\/p>\n<\/div>\n<p>Add more training data. <a href=\"https:\/\/machinethink.net\/blog\/coreml-training-part2\/\">Part 2<\/a> has links to some other datasets you can use.<\/p>\n<p>If you\u2019re up for a challenge, try adapting one of the other image classifiers from <a href=\"https:\/\/developer.apple.com\/machine-learning\/models\/\">developer.apple.com\/machine-learning\/models<\/a>, such as MobileNetV2 or ResNet50. Make them updatable and give the classifier layer 10 output neurons. How well do these train on our dataset? (Tip: You may need to change the names of the model\u2019s inputs and outputs to make them compatible with the demo app.)<\/p>\n<h3>It\u2019s a bit leaky\u2026<\/h3>\n<p>Training does take up a lot of extra RAM. This is not surprising, as for training Core ML will need to store a lot of intermediate data: the activations during the forward pass, the gradients during the backwards pass.<\/p>\n<p>In the demo app, running the <code>MLUpdateTask<\/code> takes up about 250 MB. That\u2019s not so bad, except Core ML never seems to give this memory back. Each time you start a new <code>MLUpdateTask<\/code>, it eats up another 250 MB. Yikes.<\/p>\n<p>Instruments tells me that this is all memory owned by Core ML, and it does not appear to be released ever. It\u2019s not hard to make the device run out of memory this way and crash the app. <strong>This seems like a serious bug!<\/strong><\/p>\n<p>(When you cancel the task, the memory also isn\u2019t freed. I thought maybe you\u2019re supposed to re-use <code>MLUpdateTask<\/code> objects, i.e. call <code>resume()<\/code> again on the same instance to train for another set of epochs, but this does not seem to be the case. Once the task has completed, you can\u2019t start it again.)<\/p>\n<h2>Data augmentation<\/h2>\n<p>One way to get more training data is to artifically enlarge your training set with data augmentation. The image loader is the ideal place for that.<\/p>\n<p>By randomly flipping images, you instantly get twice the amount of training data. For the purposes of demonstration, we\u2019ll add some data augmentation that does random horizontal and vertical flips. Each epoch will still have the same amount of training images, but some of these images are now mirrored.<\/p>\n<p>In <strong>ImageLoader.swift<\/strong>, the <code>featureValue(at:)<\/code> method loads the image into an <code>MLFeatureValue<\/code> object. From this you can grab the <code>CVPixelBuffer<\/code> with the image data and modify it however you like:<\/p>\n<pre><code class=\"language-swift\">func featureValue(at index: Int) throws -&gt; MLFeatureValue {\n  ...\n  let featureValue = try MLFeatureValue(imageAt: imageURL,\n                                        constraint: imageConstraint,\n                                        options: imageOptions)\n\n  if augment, let pixelBuffer = featureValue.imageBufferValue {\n    if Bool.random() {\n       flipHorizontally(pixelBuffer: pixelBuffer)\n    }\n    if Bool.random() {\n       flipVertically(pixelBuffer: pixelBuffer)\n    }\n  }\n  return featureValue\n}\n<\/code><\/pre>\n<p>We use <code>Bool.random()<\/code> to toss a coin. It this returns true, we flip the image. In case you\u2019re curious, here\u2019s how <code>flipHorizontally<\/code> is implemented:<\/p>\n<pre><code class=\"language-swift\">private func flipHorizontally(pixelBuffer: CVPixelBuffer) {\n  var image = CIImage(cvPixelBuffer: pixelBuffer)\n  image = image.transformed(by: CGAffineTransform(scaleX: -1, y: 1))\n  image = image.transformed(by: CGAffineTransform(\n                                    translationX: image.extent.width, y: 0))\n  ciContext.render(image, to: pixelBuffer)\n}\n<\/code><\/pre>\n<p>This uses Core Image to transform the image, and then renders it back into the same <code>CVPixelBuffer<\/code> object.<\/p>\n<p>After training with data augmentation enabled, the model does seem to perform a bit better on test images that are upside-down and mirrored, so I think it helped!<\/p>\n<p>Other image augmentations that are typically used are random rotations, zooming in or out, and color shifts. Usually the more augmentation, the better. Keep in mind that with a lot of data augmentation, you also need to train for enough epochs so that the model sees all the different variations of each image.<\/p>\n<h2>Training in the background<\/h2>\n<p>I was going to add a section on how to train models when the app isn\u2019t running. However, this blog post is already long enough, so I\u2019ll just mention the basics.<\/p>\n<ul>\n<li>\n<p>Training in the background isn\u2019t necessary for k-NN, as training these kinds of models is pretty much instantaneous. But it might make sense for a neural network if you have a lot of data and want to train for many epochs. Or if your model is rather big and you want to train only when the device isn\u2019t being used for anything.<\/p>\n<\/li>\n<li>\n<p>You\u2019ll need to use the new <strong>BackgroundTasks framework<\/strong>. With the classes <code>BGTaskScheduler<\/code> and <code>BGProcessingTaskRequest<\/code> you can schedule the Core ML update task to run in the background. This will wake up your app at an opportune moment and give it some processing time to run a few training epochs. I believe you can even schedule your training updates to run at night or when the device is plugged into power.<\/p>\n<\/li>\n<li>\n<p>Background training most likely uses the CPU only, as the GPU and the Neural Engine are off-limits to background processes. But in my experiments, Core ML always used the CPU for training anyway, even when the app was active.<\/p>\n<\/li>\n<\/ul>\n<p>I might do a more detailed blog post on this topic in the future.<\/p>\n<h2>The end<\/h2>\n<p>Thanks for reading all the way to the end! \ud83d\ude04<\/p>\n<p>I think it\u2019s pretty cool that we can train neural networks on the device now. Of course, it was always possible to do this by writing your own code or, since iOS 11.3, using Metal Performance Shaders. But it\u2019s very convenient that you can directly train Core ML models now!<\/p>\n<p>Training support is still a bit limited \u2014 only two loss functions, backprop doesn\u2019t work through all layer types, etc. \u2014 but no doubt more functionality will be added in the future.<\/p>\n<p>And remember, just because you can use machine learning to solve this kind of problem, doesn\u2019t mean that you should. <a href=\"https:\/\/medium.com\/@muehler.v\/simple-hand-gesture-recognition-using-opencv-and-javascript-eb3d6ced28a0\">Here is a cool blog post<\/a> that shows how to do hand gesture recognition <em>without<\/em> machine learning.<\/p>","protected":false},"excerpt":{"rendered":"<p>In this series of blog posts we\u2019re taking a deep dive into the new on-device model personalization features from Core ML 3. I\u2019ll show how to create a customizable image classifier using k-Nearest Neighbors as well as a deep neural network, right from inside an iOS app. This is the last part of a four-part [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":10202,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rop_custom_images_group":[],"rop_custom_messages_group":[],"rop_publish_now":"initial","rop_publish_now_accounts":[],"rop_publish_now_history":[],"rop_publish_now_status":"pending","footnotes":""},"categories":[1],"tags":[],"class_list":["post-10201","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-explore"],"_links":{"self":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/posts\/10201","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/comments?post=10201"}],"version-history":[{"count":0,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/posts\/10201\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/media\/10202"}],"wp:attachment":[{"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/media?parent=10201"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/categories?post=10201"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/handoli.com\/index.php\/wp-json\/wp\/v2\/tags?post=10201"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}