AVAssetWriter leading to huge kernel_task memory usage

Question

I am currently working on a macOS app which will be creating very large video files with up to an hour of content. However, generating the images and adding them to AVAssetWriter leads to VTDecoderXPCService using 16+ GB of memory and the kernel-task using 40+ GB (the max I saw was 105GB).

It seems like the generated video is not streamed onto the disk but rather written to memory for it to be written to disk all at once. How can I force it to stream the data to disk while the encoding is happening?

Btw. my app itself consistently needs around 300MB of memory, so I don't think I have a memory leak here.

Here is the relevant code:

Code Block func analyse()
{
        self.videoWritter = try! AVAssetWriter(outputURL: outputVideo, fileType: AVFileType.mp4)
        let writeSettings: [String: Any] = [
            AVVideoCodecKey: AVVideoCodecType.h264,
            AVVideoWidthKey: videoSize.width,
            AVVideoHeightKey: videoSize.height,
            AVVideoCompressionPropertiesKey: [
                AVVideoAverageBitRateKey: 10_000_000,
            ]
        ]
        self.videoWritter!.movieFragmentInterval = CMTimeMake(value: 60, timescale: 1)
        self.frameInput = AVAssetWriterInput(mediaType: AVMediaType.video, outputSettings: writeSettings)
        self.frameInput?.expectsMediaDataInRealTime = true
        self.videoWritter!.add(self.frameInput!)
        if self.videoWritter!.startWriting() == false {
            print("Could not write file: \(self.videoWritter!.error!)")
            return
        }
}
func writeFrame(frame: Frame)
{
        /* some more code to determine the correct frame presentation time stamp */
        let newSampleBuffer = self.setTimeStamp(frame.sampleBuffer, newTime: self.nextFrameStartTime!)
        self.frameInput!.append(newSampleBuffer)
        /* some more code */
}
func setTimeStamp(_ sample: CMSampleBuffer, newTime: CMTime) -> CMSampleBuffer {
        var timing: CMSampleTimingInfo = CMSampleTimingInfo(
            duration: CMTime.zero,
            presentationTimeStamp: newTime,
            decodeTimeStamp: CMTime.zero
        )
        var newSampleBuffer: CMSampleBuffer?
        CMSampleBufferCreateCopyWithNewTiming(
            allocator: kCFAllocatorDefault,
           sampleBuffer: sample,
           sampleTimingEntryCount: 1,
           sampleTimingArray: &timing,
           sampleBufferOut: &newSampleBuffer
       )
        return  newSampleBuffer!
    }

My specs:
MacBook Pro 2018
32GB Memory
macOS Big Sur 11.1

1.9k

Posted by

nbde

Reply

Add a Comment

Answer 1

The culprits were actually different ones and not in the code I posted.

1.) The kernel_task:

The following code contains the memory leak:

Code Block let croppedImage = frame.image.cropped(to: CGRect(
    x: Int(Int(videoSize.width) / 2 - (self.RECOGNITION_WIDTH / 2)),
    y: Int(self.videoSize.height - CGFloat(self.RECOGNITION_HEIGHT)),
    width: self.RECOGNITION_WIDTH,
    height: self.RECOGNITION_HEIGHT
)

Apparently CIImage.cropped(to:) leads to the original CIImage not being cleaned up. According to my Google search I should have called CGImageRelease(:) but that is not contained in Swift 5.1 since Core Foundation Objects are now memory managed. (At least that is the Xcode error message.)

So I convert the image to a CGImage, crop that one and turn it into a CIImage. I know this is not efficient, but for me that's fine. So here is the new code:

Code Block import Foundation
import CoreImage
extension CIContext
{
    func cropWithoutMemoryLeak(_ image: CIImage, to rect: CGRect) -> CIImage
    {
        let newImage = self.createCGImage(image, from: rect)
        return CIImage(cgImage: newImage!)
    }
}

2.) The VTDecoderXPCService:

I was able to reliably remove the memory leak by commenting out this line, which obviously defeats the point of creating a new video:

Code Block self.frameInput!.append(newSampleBuffer)

After trying some more stuff I removed some of my debug print statements which I had throughout my code. And all the sudden the leak was gone...? 🤷‍♀️ So I don't really know what the problem was. Removing the debug lines was most certainly a correlation but no causation. I guess it triggered a recompile of previously cached files but admittedly I have no idea, what the problem was.

Posted by

nbde

Add a Comment

Answer 2

AVAssetWriter writes frames as you append them, but it may take longer to write each frame than it takes you to append each frame. Looking at your code, you don't appear to be checking AVAssetWriterInput.readyForMoreMediaData before appending frames. Updating your code to append frames only when the input is ready may help your memory usage.

Posted by

Media Engineer

Add a Comment

Answer 3

The culprits were actually different ones and not in the code I posted.

1.) The kernel_task:

The following code contains the memory leak:

Code Block let croppedImage = frame.image.cropped(to: CGRect(
    x: Int(Int(videoSize.width) / 2 - (self.RECOGNITION_WIDTH / 2)),
    y: Int(self.videoSize.height - CGFloat(self.RECOGNITION_HEIGHT)),
    width: self.RECOGNITION_WIDTH,
    height: self.RECOGNITION_HEIGHT
)

Apparently CIImage.cropped(to:) leads to the original CIImage not being cleaned up. According to my Google search I should have called CGImageRelease(:) but that is not contained in Swift 5.1 since Core Foundation Objects are now memory managed. (At least that is the Xcode error message.)

So I convert the image to a CGImage, crop that one and turn it into a CIImage. I know this is not efficient, but for me that's fine. So here is the new code:

Code Block import Foundation
import CoreImage
extension CIContext
{
    func cropWithoutMemoryLeak(_ image: CIImage, to rect: CGRect) -> CIImage
    {
        let newImage = self.createCGImage(image, from: rect)
        return CIImage(cgImage: newImage!)
    }
}

2.) The VTDecoderXPCService:

I was able to reliably remove the memory leak by commenting out this line, which obviously defeats the point of creating a new video:

Code Block self.frameInput!.append(newSampleBuffer)

After trying some more stuff I removed some of my debug print statements which I had throughout my code. And all the sudden the leak was gone...? 🤷‍♀️ So I don't really know what the problem was. Removing the debug lines was most certainly a correlation but no causation. I guess it triggered a recompile of previously cached files but admittedly I have no idea, what the problem was.

Posted by

nbde

Add a Comment

Answer 4

I'm glad to hear you've solved or gotten around the issues you are seeing. If you haven't already, please file a bug (using the feedback assistant) about the apparent CIImage memory leak.

Posted by

Media Engineer

Add a Comment

Answer 5

I was facing this problem a lot.My CPU usage would reach above 1000 percent and most of it would be by Kernel_task. Whenever we connect second display or use heavy applications ,it requires more gpu/cpu power so our computer heats up more. If the temperature rises too high, Mac Os will run a dummy process "Kernel_task" to reserve cpu usage but keep it idle to reduce load of cpu and protect the computer from burning. As the temperature lowers it will release the resources. The problem in my case was accumulation of dust in the CPU and GPU fan.It had probably sucked it through the side vents. I cleaned it and it works like breeze. I even changed the thermal paste on my CPU and GPU as it had become hard over time and excessive heating. Hope this helps.

Posted by

RobbySingh

Add a Comment

Answer 6

Hey - a bit unrelated to the memory leak, but in my case calling AVAssetWriter.startWriting() takes around 3 seconds to complete on a modern iPhone 14 - is there any way to speed that up? See my post: https://developer.apple.com/forums/thread/741942

Thanks!

Posted by

mrousavy

Add a Comment

AVAssetWriter leading to huge kernel_task memory usage

Accepted Reply

Replies