How to visualize 16bit raw image data

Question

I'm working on a very simple App where I need to visualize an image on the screen of an iPhone. However, the image has some special properties. It's a 16bit, yuv422_yuy2 encoded image. I already have all the raw bytes saved in a Data object.

After googling for a long time, I still did not figure out the correct way. My current understanding is first create a CVPixelBuffer to properly represent the encoding information. Then conver the CVPixelBuffer to an UIImage. The following is my current implementation.

public func YUV422YUY2ToUIImage(data: Data, height: Int, width: Int, bytesPerRow: Int) -> UIImage {
    return rosImage.data.withUnsafeMutableBytes { rawPointer in
        let baseAddress = rawPointer.baseAddress!
        let tempBufferPointer = UnsafeMutablePointer<CVPixelBuffer?>.allocate(capacity: 1)
        
        CVPixelBufferCreateWithBytes( kCFAllocatorDefault,
                                      width,
                                      height,
                                      kCVPixelFormatType_422YpCbCr16,
                                      baseAddress,
                                      bytesPerRow,
                                      nil,
                                      nil,
                                      nil,
                                      tempBufferPointer)
        
        let ciImage = CIImage(cvPixelBuffer: tempBufferPointer.pointee!)
        return UIImage(ciImage: ciImage)
    }
}

However, when I execute the code, I have the followin error

-[CIImage initWithCVPixelBuffer:options:] failed because its pixel format v216 is not supported.

So it seems CIImage is unhappy. I think I need to convert the encoding from yuv422_yuy2 to something like plain ARGB. But after a long tim googling, I didn't find a way to do that. The closest function I cand find is https://developer.apple.com/documentation/accelerate/1533015-vimageconvert_422cbypcryp16toarg

But the function is too complex for me to understand how to use it.

Any help is appreciated. Thank you!

419

Posted by

yaoyuh

Reply

Add a Comment

Answer 1

I think I need to convert the encoding from yuv422_yuy2 to something like plain ARGB. But after a long tim googling, I didn't find a way to do that.

You can do it yourself. It's a few nested loops and a matrix multiplication, right?

Posted by

endecotp

I think theoretically you're right. But I think this type of operation an image processing package may already provide. Probably with proper error handling and optimization.

—
yaoyuh

Add a Comment

Answer 2

This is an accelerate based conversion for macOS but maybe it will help you along:

func normaliseImage(inImage: CGImage?) -> CGImage? {
                
        guard
            let cgImage = inImage ?? originalnsimage.cgImage(forProposedRect: nil, context: nil, hints: nil) else {return nil}
        guard
            let format = vImage_CGImageFormat(cgImage: cgImage) else {return nil}
        guard
            let sourceBuffer = try? vImage_Buffer(cgImage: cgImage, format: format) else {return nil}
        defer {sourceBuffer.free()}
                
        bitsperPixel = min(cgImage.bitsPerPixel, 32)
        bitsperComponent = min(cgImage.bitsPerComponent, 8)
        
        // Get workable argb image format in return
        let argbFormat: vImage_CGImageFormat = {
            return vImage_CGImageFormat(bitsPerComponent: bitsperComponent, bitsPerPixel: bitsperPixel, colorSpace: originalColorSpace, bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.first.rawValue), renderingIntent: .defaultIntent)!
        }()
        guard
            var destinationBuffer = try? vImage_Buffer(width: Int(sourceBuffer.width), height: Int(sourceBuffer.height), bitsPerPixel: argbFormat.bitsPerPixel) else {return nil}
        defer {destinationBuffer.free()}
        
        if let toRgbConverter = try? vImageConverter.make(sourceFormat: format, destinationFormat: argbFormat) {
            try? toRgbConverter.convert(source: sourceBuffer, destination: &destinationBuffer)
        }
        let result = try? destinationBuffer.createCGImage(format: argbFormat)
        return result
        
    }

Posted by

JosNL

Add a Comment

How to visualize 16bit raw image data

Replies