Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GPUImageMovieWriter - occasional black frames at either ends of the recorded video #1255

Open
zakinaeem opened this issue Oct 14, 2013 · 16 comments

Comments

@zakinaeem
Copy link

I have recording app implementation where user can tap the "record" button to start/stop recording. I achieve this with a basic GPUImageVideoCamera with output set to a GPUImageView as well as a GPUImageMovieWriter.

50% of the time, the recorded clip ends up with a couple (or a single) black frame at either ends, sometimes both. The implementation is fairly straightforward, but here is it anyway.

gpuImageView = [[GPUImageView alloc] initWithFrame:cameraView.frame];
gpuImageView.fillMode = kGPUImageFillModePreserveAspectRatioAndFill;

[cameraView addSubview:gpuImageView];

videoCamera = [[GPUImageVideoCamera alloc] initWithSessionPreset:AVCaptureSessionPresetHigh cameraPosition:(usingBackCamera) ? AVCaptureDevicePositionBack : AVCaptureDevicePositionFront];

[videoCamera addTarget:gpuImageView];
[videoCamera addTarget:movieWriter];
videoCamera.audioEncodingTarget = movieWriter;
[videoCamera startCameraCapture];

double delayToStartRecording = 0.5;
dispatch_time_t startTime = dispatch_time(DISPATCH_TIME_NOW, delayToStartRecording * NSEC_PER_SEC);
        dispatch_after(startTime, dispatch_get_main_queue(), ^(void){
            NSLog(@"Start recording");
            [movieWriter startRecording];
        });


And then stop the recording with the following (while the live camera continues to show up on gpuimageview.

[movieWriter finishRecording];

Has anyone else experienced this and/or found a solution to avoid black frames? I cannot pause/resume camera capture so to ensure seamless user experience.

@staykids
Copy link

I've definitely encountered this issue before. It's annoying - I spent some time trying to investigate why it occurs but it's sporadic enough that I just looked to mitigate it some other way.

Here's a low-tech solution: after the movieWriter is finished recording, start an AVAssetExportSession with the newly completely video URL, chop some "buffer time" off the video, then save it back to disk. I find that removing a CMTime duration of CMTimeMake(1,10) off both the start and end of the movieWriter video removes all black frame artifacts. YMMV. I'd recommend maybe using movieWriter finishRecordingWithCompletionHandler, and then in the completion block make a call to whatever method does the "buffer time" trimming.

If you notice, Apple's stock Camera app (when taking a video) chops off a bit of time from the beginning of a video after you finish recording. I'm not saying it's necessarily because of an issue like this, but just wanted to mention it to illustrate that it can indeed be a lot of data to deal with.

I leave it to more interested minds to discover the "why" of this issue and engineer a better solution.

@zakinaeem
Copy link
Author

Thanks for your input mate - what you suggest (chopping the frames off) is definitely in for Plan B. I never noticed the chopping off on Apple's stock camera app, interesting!

Is definitely worth finding out the reason behind this behavior and a possible solution. Will continue to look into it myself.

@m1entus
Copy link

m1entus commented Oct 20, 2013

videoCamera.audioEncodingTarget = movieWriter - this is your problem. It's very expensive operation.

@followmyheart-Bao
Copy link

I experienced the same issue, anyone else has solution?

@zmzhuai
Copy link
Contributor

zmzhuai commented Nov 27, 2013

I think I found the reason, I look into the file GPUImageVideoCamera.m, the GPUImageVideoCamera process frames, There is a semaphore named frameRenderingSemaphore, at line 951. It will wait this semaphore, when the process is not finished,It will drop the frames. and I think GPUImage does not change CMTime of CMSampleBufferRef. I do not have some solution, To solute this problem, You may rewrite this class.

@zakinaeem
Copy link
Author

No luck with this and as much as I have tried to push it back, its quite incorrect user-experience wise to have these black frames you cannot control. Anyone with a solution, please do let know.

@barrault01
Copy link

I do have the same problem and can't resolved it.

@xujingzhou
Copy link

I met the same issue. Any solution now?

@KageDev
Copy link

KageDev commented Dec 26, 2014

The problem is that the audio buffer is written before the video buffer.
Fix:

Inside GPUImageMovieWriter.m
add code:

static BOOL allowWriteAudio = NO;

- (void)startRecording;
{
  ...
  allowWriteAudio = NO;
}

- (void)processAudioBuffer:(CMSampleBufferRef)audioBuffer;
{
  if (!allowWriteAudio) {
    return;
  }
  ...
}

- (void)newFrameReadyAtTime:(CMTime)frameTime atIndex:(NSInteger)textureIndex;
{
  ...
  if (![assetWriterPixelBufferInput appendPixelBuffer:pixel_buffer withPresentationTime:frameTime])
    NSLog(@"Problem appending pixel buffer at time: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, frameTime)));

  allowWriteAudio = YES;   //< add this
  ...
}

zakinaeem added a commit to bigcitypix/GPUImage that referenced this issue Dec 26, 2014
@protez
Copy link

protez commented Jul 15, 2015

Thank you so much KageDev! You're the hero.

@yumincode
Copy link

hello,can anyOne help me?When i add this code,my video is silent,there is no sound, please help me...

hk41 pushed a commit to hk41/GPUImage that referenced this issue Apr 20, 2016
@mingyue1991
Copy link

@KageDev e~,it can avoid the black frame at the beginning, but the black frame at the end still occur sometimes.do you know how to solve it?

@jiliszc
Copy link

jiliszc commented Nov 30, 2016

5c and 4s the black frame at the end still occur

zmzhuai added a commit to zmzhuai/GPUImage that referenced this issue Dec 28, 2016
The black frame at either ends because of  audio frame is longer than video frame. Add code to guarantee video frame is longer than audio.
@zmzhuai
Copy link
Contributor

zmzhuai commented Dec 28, 2016

I added a pull request #2410 to fix the black frame issue.

@imbaZLN
Copy link

imbaZLN commented Jan 6, 2017

I've got a workaround for this issue. Basically you need to track the first video frame time and the last video frame time. After the movie writer finishes exporting the video file, trim the output file manually based on the time range from the first frame time to the last frame time.

@mingyue1991
Copy link

@zmzhuai i tried you code, great! avoid the end frame! so cool.(attention, when use movie writer do other things ,like edit video , add music , the audio will lost, so i add a var ,use this only when record video.)

ShivaHuang added a commit to ShivaHuang/GPUImage2 that referenced this issue May 30, 2017
RoCry added a commit to RoCry/GPUImage2 that referenced this issue Apr 18, 2018
lj20082 pushed a commit to lj20082/GPUImage that referenced this issue Oct 24, 2018
elhoangvu added a commit to elhoangvu/Tikky that referenced this issue Dec 10, 2018
Reason:
- The audio buffer is written before the video buffer.
Solution:
BradLarson/GPUImage#1255 (comment)
Inside GPUImageMovieWriter.m
add code:

static BOOL allowWriteAudio = NO;

- (void)startRecording;
{
  ...
  allowWriteAudio = NO;
}

- (void)processAudioBuffer:(CMSampleBufferRef)audioBuffer;
{
  if (!allowWriteAudio) {
    return;
  }
  ...
}

- (void)newFrameReadyAtTime:(CMTime)frameTime atIndex:(NSInteger)textureIndex;
{
  ...
  if (![assetWriterPixelBufferInput appendPixelBuffer:pixel_buffer withPresentationTime:frameTime])
    NSLog(@"Problem appending pixel buffer at time: %@", CFBridgingRelease(CMTimeCopyDescription(kCFAllocatorDefault, frameTime)));

  allowWriteAudio = YES;   //< add this
  ...
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests