I\u2019m using the Vision framework to run an object detection model on iOS but the predicted bounding boxes are not being drawn in the right place<\/strong>. Help!<\/p>\n<\/blockquote>\n This is a very common problem and it happens because there are several different coordinate systems that you have to translate between. Which means you need to do math!<\/p>\n In this blog post I\u2019ll explain what\u2019s going on.<\/p>\n Even though most of what I\u2019m about to explain applies even if you\u2019re not using the Vision framework, I\u2019m assuming you\u2019re using Vision to make the predictions and that you have a bunch of Using Vision to get predictions from an object detection model requires only a handful of lines of mostly boilerplate code:<\/p>\n For every detected object, you will receive one The question now is: how to correctly display this bounding box<\/strong> in your app? This is trickier than it may seem!<\/p>\n Doing object detection with Vision is only possible when the Core ML model has been configured in a particular way.<\/p>\n The model needs to be a pipeline<\/strong> that contains the actual model \u2014 such as YOLO or SSD \u2014 as well as a special NMS stage that performs non-maximum suppression. You can read more about this in a previous blog post<\/a>.<\/p>\n Not all Core ML object detection models have already been configured this way, which means you can\u2019t use them with Vision. You either need to turn the model into the kind of pipeline I just described, or write a lot of additional Swift code.<\/p>\n For example, the Ultralytics<\/a> YOLOv5<\/strong> model has a Core ML version but it requires changes<\/a> before you can use it with Vision.<\/p>\n It\u2019s useful to understand what actually happens when you give an image to Vision. The options you choose here affect how you should process the bounding boxes afterwards.<\/p>\n First, Vision will resize the image to the size expected by the model. For YOLO this is typically 416\u00d7416 pixels, for SSD it\u2019s something like 300\u00d7300 pixels.<\/p>\n Usually your input image is larger than that and it won\u2019t be square.<\/p>\n In this article I\u2019m assuming you\u2019ll be using the iPhone or iPad camera, so your image will likely be 1280\u00d7720 pixels or 1920\u00d71080 pixels.<\/p>\n So how do you get a non-square image into a square hole?<\/p>\n Vision lets you do choose an centerCrop<\/strong>: Only keep the center square and throw away everything else. For a 1920\u00d71080 image, this means the cropped image will be 1080\u00d71080 pixels. You lose 420 pixels on the left (or top) and 420 pixels on the right (or bottom). Your model won\u2019t be able to detect objects close to the edges of the image because they literally get cut off.<\/p>\n<\/li>\n scaleFit<\/strong>: Resize the image but keep the aspect ratio intact. You won\u2019t lose any pixels, but a large part of the input image will now be empty. That\u2019s not very efficient \u2014 the neural network will process those unused pixels too \u2014 and because of the large scaling factor, the image is bound to lose small details.<\/p>\n<\/li>\n scaleFill<\/strong>: Resize the image but ignore the aspect ratio. This keeps the entire image and no pixel goes unused. Unfortunately, the downside is that it will squeeze the image in one direction. For a 1920\u00d71080 image, it will have to fit 1920 pixels in the space of 1080. All the objects in the image will look stretched out now. That\u2019s not a problem if the model was trained on similarly stretched images, but usually it isn\u2019t.<\/p>\n<\/li>\n<\/ul>\n Alternatively, choose a custom crop region by setting What is the right option to choose? It depends on your app and also on how the model was trained.<\/p>\n It\u2019s important to remember that what you choose here will affect how you need to transform the bounding boxes<\/strong> later.<\/p>\n The easiest option is Note:<\/strong> Apple\u2019s docs for scaleFill<\/a> are wrong. They say, \u201cscaleFill: A scaling option that scales an image proportionally so that its shorter dimension fills the canvas, potentially cropping along the longer dimension.\u201d But this is the behavior of So far I\u2019ve assumed that the Core ML model takes a square input but this is not a requirement!<\/p>\n My advice is to make the input of the Core ML model have an aspect ratio that is close to the aspect ratio of your input images. Then you can just use With many models, such as YOLO and SSD, it\u2019s possible to change the input size without having to re-train the model from scratch, but you might have to change some of the model\u2019s internals<\/a> too (such as the anchor boxes).<\/p>\n For example, images from the camera will usually have a 16:9 aspect ratio. You might choose a model input size of 352\u00d7192 or 384\u00d7224, which have a similar aspect ratio. It\u2019s not exactly the same, but close enough. Now Note:<\/strong> For most models the input dimensions must be a multiple of 32, so you may not be able to get the aspect ratio exactly the same.<\/p>\n<\/div>\n Vision also rotates the image, if necessary.<\/p>\n The camera sensor on the iPhone is mounted in landscape orientation, and so if you\u2019re using When the device is in portrait mode, images coming from the camera are seen by the Core ML model as rotated 90 degrees to the right. That\u2019s probably not what the model expects, and it won\u2019t give good predictions!<\/p>\n Because of this, you have to tell Vision what the orientation of the image is, by passing the You can also rotate the image yourself, of course, or set the Note:<\/strong> If you\u2019re using If you\u2019re not using video at all, but regular images, you may still need to handle rotations. Images have EXIF data that contains rotation information that should be passed to Vision as well.<\/p>\n I wrote a small app for making sure that what Vision sees is indeed what you think it sees. I use this technique all the time for troubleshooting.<\/p>\n This uses a very simple Core ML model that simply outputs the same image that it was given as input. By drawing the \u201cbefore\u201d and \u201cafter\u201d images, you can see how Vision resizes \/ crops \/ rotates the image that gets passed off to Core ML.<\/p>\n The app and model are in the CheckInputImage<\/strong> folder of the Core ML Survival Guide repo<\/a>. This repo accompanies my book Core ML Survival Guide<\/a>, which you should definitely buy! \ud83d\ude04<\/p>\n To verify what Vision does to your own images, put the Image2Image.mlmodel<\/strong> into your app and run it instead of your actual model but with the same settings. Grab the output image produced by the model and draw it in a temporary Note:<\/strong> By default the Image2Image model uses 256\u00d7256 images but you can easily change this by editing the create_model.py<\/strong> script. Tip: pay attention to the content mode of the At this point, I\u2019m going to assume you set up the Vision crop \/ scale \/ orientation options properly, and that you verified the input image that Vision passed off to Core ML is correct for your model. I\u2019m also assuming that the image normalization<\/a> settings inside the model are good.<\/p>\n Whenever I troubleshoot models for my clients, I always spend a fair bit of time on making 100% sure the inputs are correct. There\u2019s no point in debugging the output of the model until you\u2019ve debugged the input!<\/p>\n The output of an object detection model is an array of bounding boxes. However, you can\u2019t \u201cjust\u201d draw these bounding boxes on top of the image.<\/p>\n Because the image that goes into your model is not necessarily the full image due to cropping done by Vision, the coordinates of the predicted bounding boxes don\u2019t \u201cmake sense\u201d unless you transform them first.<\/p>\n In addition, because the image and the screen will have different dimensions and aspect ratios, the app is usually not able to display the full image either. The image needs to be resized and\/or cropped again to make it fit the screen, and this affects the bounding boxes too.<\/p>\n What makes this tricky is that there are several different coordinate systems that we\u2019re dealing with:<\/p>\n Whenever we\u2019re talking about bounding box coordinates, it\u2019s important to understand the reference frame in which these coordinates live.<\/p>\n And to get the bounding boxes to show up in the correct place, you need to do a bit of math to convert between these different coordinate systems.<\/p>\n Here is a flowchart of the transformations that are needed:<\/p>\n For every detected object, Vision gives you a There are a few important things to know about The coordinates are normalized<\/strong>. This means x, y, width and height are all fractional numbers in between 0.0 and 1.0.<\/p>\n<\/li>\n The origin (0,0) is in the lower-left corner<\/strong>! Usually the origin is in the upper-left corner, so that\u2019s a little confusing.<\/p>\n<\/li>\n The predictions are relative to the cropped image. In other words, the origin (0,0) is not the lower-left corner of the image but of the crop region<\/strong>.<\/p>\n<\/li>\n Finally, if the image was rotated and\/or mirrored, then the bounding box coordinates are predicted on that rotated or mirrored image.<\/p>\n<\/li>\n<\/ul>\n You may not be used to dealing with normalized coordinates, but it\u2019s actually quite convenient. It makes the predictions independent of the actual pixel size of the input image. Now we don\u2019t have to worry about Core ML\u2019s coordinate system at all.<\/p>\n Note:<\/strong> Some object detection models don\u2019t use normalized coordinates for their predictions \u2014 in which case you can\u2019t use them directly with Vision. For example, if the model works on 416\u00d7416 images, the bounding box coordinates will be between 0 and 416. The easiest way to deal with such coordinates is to normalize them afterwards anyway, i.e. divide them by the image\u2019s width and height. This can be done inside the Core ML model.<\/p>\n<\/div>\n Good news: You don\u2019t<\/em> need to do this step if you pass the proper To understand what\u2019s going on here, let\u2019s say the app is in portrait mode and you\u2019ve also set the Because the camera is oriented in landscape, the Core ML model will now see an input image that is rotated by 90 degrees, and so any predictions it makes are rotated by 90 degrees too. (Assuming the model is even capable of detecting objects that are flipped on their side, because that\u2019s how it will see everything.)<\/p>\n To draw these bounding boxes on top of the A much easier solution is to write Tip:<\/strong> If your app needs to handle both portrait and landscape, change the orientation of the Important: You need to convert the For some reason, Vision\u2019s origin is in the lower-left corner. Usually we want the origin to be in the upper-left, so you\u2019ll need to flip the bounding box vertically.<\/p>\n Because the coordinates are normalized, this is easy enough: make a new You can skip this section if you used Unfortunately, there is no convenient Vision API that says: \u201cFor this mode and this image size, this is the cropping rectangle that was used.\u201d Instead, you\u2019ll have to do this math yourself.<\/p>\n My poor brain finds this easiest to do in the coordinate space of the full image, so that I can work with pixels and whole numbers, but for various reasons it\u2019s smarter to keep everything in normalized coordinates. (Other APIs that we use later on to convert to screen coordinates expect to use normalized coordinates, that\u2019s why.)<\/p>\n At this point, (0,0) is the top-left corner of the crop region and (1,1) is the bottom-right corner of the crop region.<\/p>\n What we want is (0,0) to be the top-left corner of the full image<\/strong> and (1,1) to be the bottom-right corner of the full image.<\/p>\n So we need to convert from the crop rectangle\u2019s coordinate system back to the full image\u2019s. How to do that depends on the Suppose the full image is landscape 1920\u00d71080 and you used Expressed in pixels, that means (0,0) from a bounding box really refers to pixel position (420, 0) in the full image and (1,1) refers to pixel position (1500, 1080) in the full image.<\/p>\n Or in normalized coordinates: the top-left corner of the crop region is at (0.22, 0) and the bottom-right corner of the crop region is at (0.78, 1).<\/p>\n If you\u2019re wondering where I got these numbers from: Therefore, to un-crop the bounding box, we must multiply the predicted width by The predicted x-coordinate of the bounding box must also be multiplied by the same amount. This is easy to see if you consider a bounding box with coordinate x = 1.0. This should be at the right edge of the crop region, which is at x = 0.78 in the full image. So scaling down the coordinates is necessary.<\/p>\n We must also add 0.22 to the predicted x-coordinate, because as pointed out above, the crop coordinate (0,0) is located at (0.22, 0) in the full image.<\/p>\n Pseudocode for We don\u2019t have to change the y-coordinate or the height in this example. If the image was in portrait orientation, you\u2019d use a y-offset and scale the height, and leave the x-coordinate and width unchanged.<\/p>\n The above calculation assumes that the input to the Core ML model is square. If not, the math becomes a little more complex. Let\u2019s look at how that works.<\/p>\n In Let\u2019s say the Core ML model expects a 384\u00d7256 input and your image is 1920\u00d71080.<\/p>\n In To keep the aspect ratio intact, the longest side becomes NOTE:<\/strong> Since we can\u2019t have half pixels, Vision probably trims 35 pixels off one side and 36 off the other. I have no idea how Vision actually rounds off these fractional values. Since there is no API for getting the crop rectangle, we\u2019re just guessing.<\/p>\n<\/div>\n To un-crop the bounding boxes, you have to add those pixels back. Let\u2019s do this in the normalized coordinate space again.<\/p>\n Because To un-crop the bounding box, its x-coordinate and width must be multiplied by Pseudocode for In portrait, you\u2019d use the y-coordinate and the height instead. The scale factor then is In Let\u2019s again say the Core ML model wants a 384\u00d7256 image. Now 1920 pixels becomes 384 pixels. The shortest side, 1080, becomes Note:<\/strong> These extra rows are wasted. It would have been better to use 384\u00d7224 for the model input, which only wastes 8 rows and almost has a 16:9 aspect ratio. Note that 384\u00d7216 would make the resized image fit exactly, but it\u2019s not a good choice because 216 does not evenly divide by 32 and many models require this.<\/p>\n<\/div>\n The model predicts bounding boxes in the 384\u00d7256 image, so (1,1) corresponds to position (384, 256) in this image. But because we only filled in the top 216 rows, we want (1,1) to correspond to position (384, 216).<\/p>\n We\u2019ll need to scale the prediction\u2019s y-coordinate and height by (Notice how Pseudocode for In portrait mode, you\u2019d change the width and x-coordinate instead.<\/p>\n Like I said, you don\u2019t need to do any of this when using When using Using 384\u00d7256 as the model input size \u2014 or even better 384\u00d7224 \u2014 will probably work fine with 1920\u00d71080 images and If you have a custom crop region, the math is still the mostly the same as above but in two directions instead of just one.<\/p>\n First, calculate where the corner points of the crop region are in the full image, expressed as normalized coordinates. If you used Once you have the crop region, the horizontal scaling factor is simply the width of the crop region, and the vertical scaling factor is its height.<\/p>\n Then multiply the bounding box\u2019s x-coordinate and width by the horizontal scaling factor; multiply the y-coordinate and height by the vertical scaling factor; and add the crop rect\u2019s top-left corner to the x and y-coordinates.<\/p>\n You now have the bounding boxes as normalized coordinates relative to the full 1920\u00d71080 input image. The origin (0,0) is in the top-left corner and (1,1) is in the bottom-right corner of that full image.<\/p>\n This isn\u2019t enough yet to display the bounding boxes on the screen \u2014 you still need to do the conversion from image to view coordinates (see the next section).<\/p>\n At this point, you may want to leave the bounding boxes in normalized form, or you can convert the normalized coordinates to pixels, so that they are in the coordinate space of your full image.<\/p>\n Which one you should do depends on what API you plan to use for the conversion to view coordinates. Several of these APIs actually want normalized coordinates, in which case you\u2019d skip this step.<\/p>\n Sometimes I find it useful to write a \u201cdebug\u201d video that shows the predictions to a movie file, so I can download that movie from the phone and examine it in detail later. In that case, I\u2019d use Core Graphics to draw the bounding boxes directly onto the original video frame and then write the frame to the movie file. It makes sense to have the bounding boxes in pixel coordinates for this.<\/p>\n Let\u2019s say you decide to convert from normalized coordinates to pixels. This is easy enough now that you\u2019ve undone the cropping: just multiply the coordinates by the width and height of the full image. Done!<\/p>\n Vision has a handy Note:<\/strong> You may be wondering, if the image is 1920\u00d71080, is the normalized coordinate (1,1) equal to the pixel at (1920, 1080) or (1919, 1079)? Good question. This depends on how you interpret where the center of a pixel is. I don\u2019t worry too much about it \u2014 the predicted bounding boxes are not going to be pixel-perfect anyway. No one will notice if you draw them slightly too large or too small.<\/p>\n<\/div>\n Assuming you want to display the bounding boxes on the screen, they will end up in some kind of Regardless of how the results will be displayed, it\u2019s safe to say that the image or camera frame you\u2019re working with will be larger (or smaller) than the number of pixels on the screen. And usually you\u2019ll capture video in a 16:9 or 4:3 aspect ratio but most iOS devices do not have screens in that aspect ratio.<\/p>\n For example, the image may be 1080\u00d71920 pixels (portrait) while your phone\u2019s screen is perhaps 390\u00d7844 points. We can\u2019t resize the image 1:1 to fit it into the view \u2014 it would look very distorted.<\/p>\n Usually you\u2019d want to resize the video while keeping the aspect ratio<\/strong> intact. This can be done by adding \u201cletterboxing\u201d on the sides of the video or by having some portions of the frame fall outside the visible screen.<\/p>\n This is very similar to the resizing and cropping that happens when the image gets passed into the Core ML model, except this time it\u2019s for displaying the image on the screen inside a view of some kind.<\/p>\n The bounding box coordinates need to be transformed in the same way. This final translation step turns the (normalized) image coordinates into view coordinates.<\/p>\n There are many options here, depending on what API you\u2019re using to display the images:<\/p>\n If you\u2019re using ARKit has its own way to crop and scale the video frames. You can use the method ![\"Cute](\"https:\/\/machinethink.net\/images\/bounding-boxes\/cat@2x.jpg\")
<\/a>
\n<\/figure>\nVision and VNRecognizedObjectObservation<\/h2>\n
VNRecognizedObjectObservation<\/code> objects that you want to draw on the screen.<\/p>\n\/\/ Tell Core ML to use the Neural Engine if available.\nlet config = MLModelConfiguration()\nconfig.computeUnits = .all\n\n\/\/ Load the Core ML model.\nlet coreMLModel = try YourModel(configuration: config)\n\n\/\/ Create a Vision wrapper for the Core ML model.\nlet visionModel = try VNCoreMLModel(for: coreMLModel.model)\n\n\/\/ Tell Vision which model input receives the image.\n\/\/ The model may also have additional inputs for the IOU and\n\/\/ confidence thresholds used by the NMS stage.\nvisionModel.inputImageFeatureName = \"image\"\n\n\/\/ The request object and its completion handler that gets\n\/\/ called when the request completes.\nlet visionRequest = VNCoreMLRequest(model: visionModel) { request, error\n if let results = request.results as? [VNRecognizedObjectObservation] {\n \/* do stuff with the observations *\/\n }\n}\n\n\/\/ How Vision will resize and\/or crop the image data.\nrequest.imageCropAndScaleOption = .scaleFill\n\n\/\/ Run the model on an image or CVPixelBuffer.\nlet handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, \n orientation: .up)\ntry handler.perform([visionRequest])\n<\/code><\/pre>\nVNRecognizedObjectObservation<\/code>. This gives you the bounding box for the object, as well as any predicted class labels.<\/p>\nBut this only works if\u2026<\/h3>\n
How does Vision treat the image?<\/h2>\n
Resize and crop<\/h3>\n
VNImageCropAndScaleOption<\/code> that determines how the image gets shrunken down to the model\u2019s input size.<\/p>\n\n
regionOfInterest<\/code> on the VNImageBasedRequest<\/code>. Of course, you can always do the cropping yourself too.<\/p>\n<\/a>
\n<\/figure>\nscaleFill<\/code> because then you don\u2019t have to do anything special later on. But it will distort your image if the aspect ratios are very different. With centerCrop<\/code>, scaleFit<\/code>, or a custom crop rect, you will have to do additional math afterwards to \u201cuncrop\u201d the bounding boxes.<\/p>\ncenterCrop<\/code>, not scaleFill<\/code>.<\/p>\n<\/div>\nDoes it have to be square?<\/h3>\n
scaleFill<\/code> and not worry about the cropping at all.<\/p>\nscaleFill<\/code> resizing will not stretch the image \u2014 at least not to the extent you\u2019d notice.<\/p>\nIs the image in the correct orientation?<\/h3>\n
AVCaptureSession<\/code>, the CVPixelBuffer<\/code>s are also in landscape. That\u2019s fine if the app is also in landscape mode, but not when it\u2019s in portrait mode.<\/p>\norientation<\/code> argument to VNImageRequestHandler<\/code>. For video frames from AVCaptureSession<\/code>, this is .right<\/code>, which tells Vision that it should rotate the image 90 degrees to put it the proper way round.<\/p>\nvideoOrientation<\/code> property of the AVCaptureVideoDataOutput<\/code> to .portrait<\/code>. In this case, you don\u2019t need to pass an orientation to Vision, or use the default .up<\/code> orientation.<\/p>\nAVCaptureVideoPreviewLayer<\/code> to display the live camera feed in your app, that has its own videoOrientation<\/code>. Just setting that to .portrait<\/code> is not enough to rotate the actual CVPixelBuffer<\/code>s that are being captured, it only affects the preview layer.<\/p>\n<\/div>\nTip: verify what Vision sees<\/h3>\n
<\/a>
\n<\/figure>\nUIImageView<\/code>. This is literally the same thing Core ML will see when it runs your model.<\/p>\nUIImageView<\/code>. This affects how the image is being displayed, and may therefore be misleading. Make sure the image view is large enough to fit the entire image. Choose \u201cTop Left\u201d for the content mode, so the image does not get scaled by the UIImageView<\/code>.<\/p>\n<\/div>\nHow to convert the bounding boxes<\/h2>\n
\n
<\/a>
\n<\/figure>\nVision outputs normalized coordinates<\/h3>\n
VNRecognizedObjectObservation<\/code>. Its superclass, VNDetectedObjectObservation<\/code>, has a boundingBox<\/code> property that is a CGRect<\/code> with the bounding box coordinates of the detected object.<\/p>\nboundingBox<\/code>:<\/p>\n\n
Un-rotating the bounding boxes<\/h3>\n
orientation<\/code> to VNImageRequestHandler<\/code>.<\/p>\nAVCaptureVideoPreviewLayer<\/code> to .portrait<\/code>. You left the orientation of the AVCaptureVideoDataOutput<\/code>\u2019s AVCaptureConnection<\/code> as .landscapeRight<\/code> (the default) and you used orientation: .up<\/code> in the VNImageRequestHandler<\/code>.<\/p>\nAVCaptureVideoPreviewLayer<\/code>, you will have to rotate them back 90 degrees the other way first. Yuck.<\/p>\nVNImageRequestHandler(..., orientation: .right)<\/code> so that Vision already fixes the image\u2019s rotation before passing it to Core ML. Now Core ML sees the image with the proper side up, the predicted bounding boxes will also be in the correct orientation already, and you don\u2019t need to worry about rotations anymore.<\/p>\nAVCaptureVideoPreviewLayer<\/code> upon any device rotations but keep the AVCaptureConnection<\/code> in landscape always. Pass the UIDevice<\/code> orientation to VNImageRequestHandler<\/code>, so Vision knows by how much to rotate the image.<\/p>\nUIDeviceOrientation<\/code> into something that matches the AVCaptureConnection<\/code> orientation. For example, if the connection is .landscapeRight<\/code> and the device is .landscapeLeft<\/code> (!!!), the Vision orientation should be .up<\/code>. For some reason, left and right are mixed up here, but this is the natural camera orientation. For device .landscapeRight<\/code>, the Vision orientation should be .down<\/code>. If the device is in portrait, the Vision orientation should be .right<\/code> as explained above.<\/p>\n<\/div>\nFixing Vision\u2019s weird origin<\/h3>\n
CGRect<\/code> with the y-coordinate at 1 - boundingBox.origin.y<\/code>. Now (0,0) is in the top-left corner again.<\/p>\nUndoing the cropping<\/h3>\n
.scaleFill<\/code>. However, for .centerCrop<\/code>, .scaleFit<\/code>, or custom crop rects you will<\/em> need to do this.<\/p>\nimageCropAndScaleOption<\/code> you chose.<\/p>\ncenterCrop<\/h4>\n
.centerCrop<\/code>, then Vision cropped out the 1080\u00d71080 square from the center of the image and made its predictions on that.<\/p>\n420\/1920 = 0.22<\/code> and 1500\/1920 = 0.78<\/code> (rounded to two decimals).<\/p>\n0.78 - 0.22 = 0.56<\/code> because the crop region\u2019s width is 56% of the full image.<\/p>\ncenterCrop<\/code> mode, for a landscape image:<\/p>\nlet scaleX = fullHeight \/ fullWidth \/\/ e.g. 1080 \/ 1920\nlet offsetX = (1 - scaleX) \/ 2\n\nvar rect = observation.boundingBox\nrect.origin.x *= scaleX\nrect.size.width *= scaleX\nrect.origin.x += offsetX\n<\/code><\/pre>\ncenterCrop (when crop region is not square)<\/h4>\n
.centerCrop<\/code> mode, Vision scales the image so that the shortest side fills up the image, while keeping the aspect ratio intact. Then it chops off the excess pixels on the other side.<\/p>\n.centerCrop<\/code> mode, it first resizes the image so that 1080 becomes 256 pixels.<\/p>\n1920 \u00d7 256\/1080 = 455<\/code> pixels wide. This is wider than 384, so it chops off (455 - 384)\/2 = 35.5<\/code> pixels on either side to make the resized image exactly 384\u00d7256.<\/p>\n35.5\/455 = 0.078<\/code> and (35.5 + 384)\/455 = 0.922<\/code>, the top-left corner of the crop region is now at (0.078, 0) and the bottom-right corner is at (0.922, 1).<\/p>\n0.922 - 0.078 = 0.844<\/code> and we must add 0.078<\/code> to the x-coordinate.<\/p>\ncenterCrop<\/code> mode in case the model input is not square, for a landscape image:<\/p>\n\/\/ (1080 \u00d7 384) \/ (1920 \u00d7 256)\nlet scaleX = (fullHeight * inputWidth) \/ (fullWidth * inputHeight) \nlet offsetX = (1 - scaleX) \/ 2\n\nvar rect = observation.boundingBox\nrect.origin.x *= scaleX\nrect.size.width *= scaleX\nrect.origin.x += offsetX\n<\/code><\/pre>\nlet scaleY = (fullWidth * inputHeight) \/ (fullHeight * inputWidth)<\/code>.<\/p>\naspectFit<\/h4>\n
.aspectFit<\/code> mode, the math is different. Vision now resizes the image so that the longest size fits, while keeping the aspect ratio intact. Then it adds padding on the shortest side to fill up the rest of the image.<\/p>\n1080 \u00d7 384\/1920 = 216<\/code> pixels. That means 256 - 216 = 40<\/code> rows at the bottom of the image will be filled with zeros.<\/p>\n216\/256 = 0.844<\/code>. There\u2019s no need to add an offset to the y-coordinate.<\/p>\n384\/455<\/code> from the previous section and 216\/256<\/code> from this section are both 0.844<\/code>. That\u2019s no coincidence.)<\/p>\naspectFit<\/code> mode, for a landscape image:<\/p>\n\/\/ (1080 \u00d7 384) \/ (1920 \u00d7 256)\nlet scaleY = (fullHeight * inputWidth) \/ (fullWidth * inputHeight) \n\nvar rect = observation.boundingBox\nrect.origin.y *= scaleY\nrect.size.height *= scaleY\n<\/code><\/pre>\nscaleFill<\/h4>\n
.scaleFill<\/code>. That\u2019s why I prefer using it, because it\u2019s the lazy<\/del>smart person\u2019s choice.<\/p>\n.scaleFill<\/code>, you do have to make sure the input size used by the model has an aspect ratio that is close to your image\u2019s actual aspect ratio or else the image will get unrealistically stretched and the model\u2019s predictions won\u2019t be great.<\/p>\n.scaleFill<\/code>.<\/p>\nCustom crop rect<\/h4>\n
regionOfInterest<\/code> to set the crop rectangle, you already have this (although it\u2019s flipped vertically).<\/p>\nlet cropRect = ... \/\/ normalized coordinates\n\nlet scaleX = cropRect.size.width\nlet scaleY = cropRect.size.height\n\nvar rect = observation.boundingBox\nrect.origin.x *= scaleX\nrect.origin.y *= scaleY\nrect.size.width *= scaleX\nrect.size.height *= scaleY\n\nrect.origin.x += cropRect.origin.x\nrect.origin.y += cropRect.origin.y\n<\/code><\/pre>\nConverting to pixel coordinates<\/h3>\n
VNImageRectForNormalizedRect()<\/code> function for this.<\/p>\nConverting to view coordinates<\/h3>\n
UIView<\/code> or CALayer<\/code>. This view may cover the entire screen, or perhaps it only takes up a small portion.<\/p>\n<\/a>
\n<\/figure>\n\n
AVCaptureVideoPreviewLayer<\/code>, the videoGravity<\/code> determines how the pixels from the camera frame are shown on the screen. This is very similar to the imageCropAndScaleOption<\/code>: there are .resizeAspect<\/code> and .resizeAspectFill<\/code> modes that determine whether the image will fill up the entire screen or whether letterboxing will be added.<\/p>\nAVCaptureVideoPreviewLayer<\/code> has a layerRectConverted(...)<\/code> method that can convert normalized coordinates to a rectangle inside the preview layer\u2019s coordinate system.<\/p>\n<\/li>\ndisplayTransform(for:viewportSize:)<\/code> from