IOS下利用OpenCV框架去除视频水印
想做个去水印的APP，第一个想到的就是CV里的inpaint图像修复技术。就想着把CV框架放在IOS中用，由于第一次接触IOS的开发，就看了两本实习时候导师大神推荐的书，很多东西都不太了解，虽然CV官网给了IOS的frame框架但做的过程中还是遇到不少坑，并且网上关于在IOS下使用OpenCV进行图像处理的资料比较少，所以就记录下做的过程中爬过的一些坑，一来由于经常搬CSND的砖，也算为D友们做点贡献。二来也是记录下自己，不然一段时间又全忘完了。由于是互联网小白纸，基本上都是靠着****和GitHub各种搬砖，有写错的地方还烦请各位大佬指出来便于成长，感激不尽，话不多说上硬货。
一、OpenCV框架集成
方法1，在OpenCV官网的下载IOS pack，直接将framework拖入项目；
方法2，使用CocoaPods直接集成OpenCV。
两种方法都可以顺利集成OpenCV框架，但在编译的时候遇到了第一个坑！
坑：提示了一堆文件not found
解决方法：opencv需要执行c++编译，xcode的.m文件并不支持相应的编译条件，需要将所有直接或者间接用到opencv头文件的所有.m文件变成.mm文件后缀，告诉编译器是oc++文件执行编译。
这个坑如果不处理好是很坑的...因为所有直接或者间接用到opencv头文件的.m文件都得改文件后缀，就会形成连锁反应导致所有的.m文件都得改掉，所以正确的方法应该是将仅利用opencv的实现.mm文件进行封装，仅保留.h的接口提供外部使用。
二、图像格式转换
OpenCV的图像通用格式是Mat类型，而在IOS中经常直接使用的图像类型是UIImage，这就牵扯到图像类型转换的问题，对于类型转换官网给了直接的Demo可以使用：
-(cv::Mat)CVMat:(UIImage*)imageSrc
{
CGColorSpaceRef colorSpace = CGImageGetColorSpace(imageSrc.CGImage);
CGFloat cols = imageSrc.size.width;
CGFloat rows = imageSrc.size.height;
cv::Mat cvMat(rows, cols, CV_8UC4); // 8 bits per component, 4 channels
CGContextRef contextRef = CGBitmapContextCreate(cvMat.data, // Pointer to backing data
cols, // Width of bitmap
rows, // Height of bitmap
8, // Bits per component
cvMat.step[0], // Bytes per row
colorSpace, // Colorspace
kCGImageAlphaNoneSkipLast |
kCGBitmapByteOrderDefault); // Bitmap info flags
CGContextDrawImage(contextRef, CGRectMake(0, 0, cols, rows), imageSrc.CGImage);
CGContextRelease(contextRef);
return cvMat;
}
-(UIImage *)UIImageFromCVMat:(cv::Mat)cvMat
{
NSData *data = [NSData dataWithBytes:cvMat.data length:cvMat.elemSize()*cvMat.total()];
CGColorSpaceRef colorSpace;
if (cvMat.elemSize() == 1) {
colorSpace = CGColorSpaceCreateDeviceGray();
} else {
colorSpace = CGColorSpaceCreateDeviceRGB();
}
CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef)data);
// Creating CGImage from cv::Mat
CGImageRef imageRef = CGImageCreate(cvMat.cols, //width
cvMat.rows, //height
8, //bits per component
8 * cvMat.elemSize(), //bits per pixel
cvMat.step[0], //bytesPerRow
colorSpace, //colorspace
kCGImageAlphaNone|kCGBitmapByteOrderDefault,// bitmap info
provider, //CGDataProviderRef
NULL, //decode
false, //should interpolate
kCGRenderingIntentDefault //intent
);
UIImage *finalImage = [UIImage imageWithCGImage:imageRef];
CGImageRelease(imageRef);
CGDataProviderRelease(provider);
CGColorSpaceRelease(colorSpace);
return finalImage;
}
但在这里使用时就遇到了第二个坑！直接使用这两个函数对一个UIImage进行处理，即先用(cv::Mat)CVMat:(UIImage*)imageSrc转换成Mat，再用(UIImage *)UIImageFromCVMat:(cv::Mat)cvMat，你会发现转换后的图像显示时跟转换前的图像方向发生了变化。分析后发现因为转换函数使用了CGImage进行类型转换，而IOS中的CGImage的坐标系跟UIImage的坐标系不一致导致的，所以可以在执行转换成Mat的函数里先根据图像方向进行旋转校正，校正函数如下：
- (UIImage *)fixOrientation {
// No-op if the orientation is already correct
if (self.imageOrientation == UIImageOrientationUp) return self;
// We need to calculate the proper transformation to make the image upright.
// We do it in 2 steps: Rotate if Left/Right/Down, and then flip if Mirrored.
CGAffineTransform transform = CGAffineTransformIdentity;
switch (self.imageOrientation) {
case UIImageOrientationDown:
case UIImageOrientationDownMirrored:
transform = CGAffineTransformTranslate(transform, self.size.width, self.size.height);
transform = CGAffineTransformRotate(transform, M_PI);
break;
case UIImageOrientationLeft:
case UIImageOrientationLeftMirrored:
transform = CGAffineTransformTranslate(transform, self.size.width, 0);
transform = CGAffineTransformRotate(transform, M_PI_2);
break;
case UIImageOrientationRight:
case UIImageOrientationRightMirrored:
transform = CGAffineTransformTranslate(transform, 0, self.size.height);
transform = CGAffineTransformRotate(transform, -M_PI_2);
break;
case UIImageOrientationUp:
case UIImageOrientationUpMirrored:
break;
}
switch (self.imageOrientation) {
case UIImageOrientationUpMirrored:
case UIImageOrientationDownMirrored:
transform = CGAffineTransformTranslate(transform, self.size.width, 0);
transform = CGAffineTransformScale(transform, -1, 1);
break;
case UIImageOrientationLeftMirrored:
case UIImageOrientationRightMirrored:
transform = CGAffineTransformTranslate(transform, self.size.height, 0);
transform = CGAffineTransformScale(transform, -1, 1);
break;
case UIImageOrientationUp:
case UIImageOrientationDown:
case UIImageOrientationLeft:
case UIImageOrientationRight:
break;
}
// Now we draw the underlying CGImage into a new context, applying the transform
// calculated above.
CGContextRef ctx = CGBitmapContextCreate(NULL, self.size.width, self.size.height,
CGImageGetBitsPerComponent(self.CGImage), 0,
CGImageGetColorSpace(self.CGImage),
CGImageGetBitmapInfo(self.CGImage));
CGContextConcatCTM(ctx, transform);
switch (self.imageOrientation) {
case UIImageOrientationLeft:
case UIImageOrientationLeftMirrored:
case UIImageOrientationRight:
case UIImageOrientationRightMirrored:
// Grr...
CGContextDrawImage(ctx, CGRectMake(0,0,self.size.height,self.size.width), self.CGImage);
break;
default:
CGContextDrawImage(ctx, CGRectMake(0,0,self.size.width,self.size.height), self.CGImage);
break;
}
// And now we just create a new UIImage from the drawing context
CGImageRef cgimg = CGBitmapContextCreateImage(ctx);
UIImage *img = [UIImage imageWithCGImage:cgimg];
CGContextRelease(ctx);
CGImageRelease(cgimg);
return img;
}
从类型转换看，UIImage转到CV:Mat丧失了一个很重要的信息，就是UIImage的imageOrientation属性。所以最好在函数的调用时能将这个参数保存下来便于后续使用。
三、inpaint图像修复技术
OpenCV提供了inpaint图像修复算法，可以进行简单的水印消除处理，该算法的原理可以参考这篇博文，这里在调用inpaint函数时遇到了第三个坑！
首先看下inpaint函数参数：
CV_EXPORTS_W void inpaint( InputArray src, InputArray inpaintMask, OutputArray dst, double inpaintRadius, int flags );
其中
InputArray src 表示要修复的图像，Mat类型
InputArray inpaintMask表示修复蒙版，Mat类型，单通道与src size一致，需要修复的地方为非0值，不修复的地方为0值
OutputArray dst 表示修复后的图像，Mat类型，与src size一致
double inpaintRadius 表示修复的半径，
int flags 表示修复使用的算法 。 opencv提供了两种选择 CV_INPAINT_TELEA 和 CV_INPAINT_NS。两种方法效果都不错。
但是在执行的时候就会报错，这是因为inpaint函数只支持3通道图像的处理，但是我们回看下官网给出的UIImage转Mat函数中是定义的4通道： cv::Mat cvMat(rows, cols, CV_8UC4); // 8 bits per component, 4 channels
我是不太愿意将这个4通道直接改成3通道使用的，因为毕竟官网给的demo能不动还是不要动，怕衍生其他的问题。。所以我采用在使用前执行一个4通道转3通道的转换：
//通道转换
IplImage imageIpl = imageMat;
IplImage *img3chan = cvCreateImage(cvGetSize(&imageIpl),imageIpl.depth,3);
cvCvtColor(&imageIpl,img3chan,CV_RGBA2RGB);//CV_RGBA2RGB表示4通道转成3通道
CvMat *cvMat = cvCreateMat(imageIpl.height, imageIpl.width, CV_8UC3);//创建容器区域
cvConvert(img3chan, cvMat);
cv::Mat imageMat3chan = cv::cvarrToMat(cvMat);
这样就可以顺利执行inpaint函数了，执行完后记得将Mat型转换成UIImage进行输出显示就好。
四、IOS中的音视频分解与合成
处理视频的核心算法就是利用OpenCV的inpaint函数，前面已经通过了。接下来就是如何把手机相册里调出来的视频进行帧分解，帧处理后与原语音进行合成形成新的视频，我这里利用IOS的AVFoundation框架进行视频处理，直接移植大神的代码真是爽。
1，帧分解：
fileUrl是视频URL地址，fps是帧率，建议写30，因为手机视频一般帧率都是30帧/秒，分解后的图像都存在splitimgs里，都为Uiimage格式。
-(void)splitVideo:(NSURL*)fileUrl fps:(float)fps splitCompleteBlock:(void(^)(BOOL success, NSMutableArray *splitimgs))splitCompleteBlock {
if (!fileUrl) {
return;
}
NSMutableArray *splitImages = [NSMutableArray array];
NSDictionary *optDict = [NSDictionary dictionaryWithObject:[NSNumber numberWithBool:NO] forKey:AVURLAssetPreferPreciseDurationAndTimingKey];
AVURLAsset *avasset = [[AVURLAsset alloc] initWithURL:fileUrl options:optDict];
CMTime cmtime = avasset.duration; //视频时间信息结构体
Float64 durationSeconds = CMTimeGetSeconds(cmtime); //视频总秒数
NSMutableArray *times = [NSMutableArray array];
Float64 totalFrames = durationSeconds * fps; //获得视频总帧数
CMTime timeFrame;
for (int i = 1; i <= totalFrames; i++) {
timeFrame = CMTimeMake(i, fps); //第i帧 帧率
NSValue *timeValue = [NSValue valueWithCMTime:timeFrame];
[times addObject:timeValue];
}
AVAssetImageGenerator *imgGenerator = [[AVAssetImageGenerator alloc] initWithAsset:avasset]; //防止时间出现偏差
imgGenerator.requestedTimeToleranceBefore = kCMTimeZero;
imgGenerator.requestedTimeToleranceAfter = kCMTimeZero;
NSInteger timesCount = [times count]; // 获取每一帧的图片
[imgGenerator generateCGImagesAsynchronouslyForTimes:times completionHandler:^(CMTime requestedTime, CGImageRef _Nullable image, CMTime actualTime, AVAssetImageGeneratorResult result, NSError * _Nullable error) {
NSLog(@"current-----: %lld", requestedTime.value);
NSLog(@"timeScale----: %d",requestedTime.timescale); // 帧率
BOOL isSuccess = NO;
switch (result) {
case AVAssetImageGeneratorCancelled:
NSLog(@"Cancelled");
break;
case AVAssetImageGeneratorFailed:
NSLog(@"Failed");
break;
case AVAssetImageGeneratorSucceeded: {
UIImage *frameImg = [UIImage imageWithCGImage:image];
UIImage *frameFit = [self reSizeImage:frameImg toSize:CGSizeMake((int)frameImg.size.width - (int)frameImg.size.width%16, (int)frameImg.size.height - (int)frameImg.size.height%16)];
[splitImages addObject:frameFit];
if (requestedTime.value == timesCount) {
isSuccess = YES;
NSLog(@"completed");
}
}
break;
}
if (splitCompleteBlock) {
splitCompleteBlock(isSuccess,splitImages);
}
}];
}
在这里也遇到一个小坑，就是在后面合成完了会发现有的视频出现绿边，这是因为AVFoundation框架里对图像的长宽有约束，必须要是16的倍数，否则会自动补绿边，所以我在函数的输出时对图像的长宽进行了处理，保证是16的倍数：
UIImage *frameFit = [self reSizeImage:frameImg toSize:CGSizeMake((int)frameImg.size.width - (int)frameImg.size.width%16, (int)frameImg.size.height - (int)frameImg.size.height%16)];
- (UIImage *)reSizeImage:(UIImage *)image toSize:(CGSize)reSize
{
UIGraphicsBeginImageContext(CGSizeMake(reSize.width, reSize.height));
[image drawInRect:CGRectMake(0, 0, reSize.width, reSize.height)];
UIImage *reSizeImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
return reSizeImage;
}
2，多张图片形成视频：
处理完后的一张张图像需要变成视频输出，也是靠着D友大神的代码，这里注意下输出的帧率与分解的帧率保持一致，均为30帧/秒
-(void)fromPicsToVideo:(NSMutableArray *)imageArray{
//设置mov路径
NSArray *paths =NSSearchPathForDirectoriesInDomains(NSCachesDirectory,NSUserDomainMask,YES);
NSString *moviePath =[[paths objectAtIndex:0]stringByAppendingPathComponent:[NSString stringWithFormat:@"%@.mp4",@"temp"]];
self.picsTovideoPath = [NSURL fileURLWithPath:moviePath];
NSFileManager* fileManager=[NSFileManager defaultManager];
if ([fileManager fileExistsAtPath:moviePath]) {
NSLog(@" have");
BOOL blDele= [fileManager removeItemAtPath:moviePath error:nil];
if (blDele) {
NSLog(@"dele success");
}else {
NSLog(@"dele fail");
}
}
//定义视频的大小
CGSize size =CGSizeMake(self.image.size.width,self.image.size.height);
NSError *error =nil;
// 转成UTF-8编码
unlink([moviePath UTF8String]);
NSLog(@"path->%@",moviePath);
// iphone提供了AVFoundation库来方便的操作多媒体设备，AVAssetWriter这个类可以方便的将图像和音频写成一个完整的视频文件
AVAssetWriter *videoWriter = [[AVAssetWriter alloc] initWithURL:[NSURL fileURLWithPath:moviePath] fileType:AVFileTypeQuickTimeMovie error:&error];
NSParameterAssert(videoWriter);
if(error)
NSLog(@"error =%@", [error localizedDescription]);
//mov的格式设置 编码格式 宽度 高度
NSDictionary *videoSettings =[NSDictionary dictionaryWithObjectsAndKeys:AVVideoCodecTypeH264,AVVideoCodecKey,
[NSNumber numberWithInt:size.width],AVVideoWidthKey,
[NSNumber numberWithInt:size.height],AVVideoHeightKey,nil];
AVAssetWriterInput *writerInput =[AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:videoSettings];
NSDictionary*sourcePixelBufferAttributesDictionary =[NSDictionary dictionaryWithObjectsAndKeys:[NSNumber numberWithInt:kCVPixelFormatType_32ARGB],kCVPixelBufferPixelFormatTypeKey,nil];
// AVAssetWriterInputPixelBufferAdaptor提供CVPixelBufferPool实例,
// 可以使用分配像素缓冲区写入输出文件。使用提供的像素为缓冲池分配通常
// 是更有效的比添加像素缓冲区分配使用一个单独的池
AVAssetWriterInputPixelBufferAdaptor *adaptor =[AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:writerInput sourcePixelBufferAttributes:sourcePixelBufferAttributesDictionary];
NSParameterAssert(writerInput);
NSParameterAssert([videoWriter canAddInput:writerInput]);
if ([videoWriter canAddInput:writerInput])
{
NSLog(@"videoWriter canAddInput:writerInput");
}
else
{
NSLog(@"videoWriter cannotAddInput:writerInput");
}
[videoWriter addInput:writerInput];
[videoWriter startWriting];
[videoWriter startSessionAtSourceTime:kCMTimeZero];
//合成多张图片为一个视频文件
int total_frame = second * 30;
int frames = (int)self.movieAsset.duration.value;
int step = frames/total_frame;
dispatch_queue_t dispatchQueue =dispatch_queue_create("mediaInputQueue",NULL);
int __block frame =0;
[writerInput requestMediaDataWhenReadyOnQueue:dispatchQueue usingBlock:^{
while([writerInput isReadyForMoreMediaData])
{
if(++frame >=[imageArray count] * step)
{
[writerInput markAsFinished];
[videoWriter finishWritingWithCompletionHandler:^(){
NSLog (@"finished writing");
dispatch_async(dispatch_get_main_queue(), ^{
[self.progressView setProgress:0.75];
[self addAudioToVideo:self.srcAudioTrack videoURL:self.picsTovideoPath];
});
}];
break;
}
CVPixelBufferRef buffer =NULL;
int idx =frame / step;
NSLog(@"idx==%d",idx);
buffer = (CVPixelBufferRef)[self pixelBufferFromCGImage:[[imageArray objectAtIndex:idx] CGImage] size:size];
if (buffer)
{
if(![adaptor appendPixelBuffer:buffer withPresentationTime:CMTimeMake(frame,600)])//设置每秒钟播放图片的个数
{
NSLog(@"FAIL");
}
else
{
NSLog(@"OK");
}
CFRelease(buffer);
}
}
}];
}
3，音视频合成
最后就是将原视频的音频与生成的视频进行合成，形成视频输出，这里都有现成的代码就直接贴了，注意开始的时候把原视频的音轨保存好就行。
-(void)addAudioToVideo:(AVAssetTrack*)srcAudioTrack videoURL:(NSURL*)videoURL{
// mbp提示框
//[MBProgressHUD showMessage:@"正在处理中"];
// 路径
NSArray *paths =NSSearchPathForDirectoriesInDomains(NSCachesDirectory,NSUserDomainMask,YES);
NSString *outPutFilePath =[[paths objectAtIndex:0]stringByAppendingPathComponent:[NSString stringWithFormat:@"%@.mp4",@"merge"]];
NSFileManager* fileManager=[NSFileManager defaultManager];
if ([fileManager fileExistsAtPath:outPutFilePath]) {
NSLog(@" have");
BOOL blDele= [fileManager removeItemAtPath:outPutFilePath error:nil];
if (blDele) {
NSLog(@"dele success");
}else {
NSLog(@"dele fail");
}
}
// 添加合成路径
NSURL *outputFileUrl = [NSURL fileURLWithPath:outPutFilePath];
// 时间起点
CMTime nextClistartTime = kCMTimeZero;
// 创建可变的音视频组合
AVMutableComposition *comosition = [AVMutableComposition composition];
// 视频采集
AVURLAsset *videoAsset = [[AVURLAsset alloc] initWithURL:self.picsTovideoPath options:nil];
// 视频时间范围
CMTimeRange videoTimeRange = CMTimeRangeMake(kCMTimeZero, videoAsset.duration);
// 视频通道 枚举 kCMPersistentTrackID_Invalid = 0
AVMutableCompositionTrack *videoTrack = [comosition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
// 视频采集通道
AVAssetTrack *videoAssetTrack = [[videoAsset tracksWithMediaType:AVMediaTypeVideo] firstObject];
// 把采集轨道数据加入到可变轨道之中
[videoTrack insertTimeRange:videoTimeRange ofTrack:videoAssetTrack atTime:nextClistartTime error:nil];
//声音采集
// 因为视频短这里就直接用视频长度了,如果自动化需要自己写判断
CMTimeRange audioTimeRange = videoTimeRange;
// 音频通道
AVMutableCompositionTrack *audioTrack = [comosition addMutableTrackWithMediaType:AVMediaTypeAudio preferredTrackID:kCMPersistentTrackID_Invalid];
// 加入合成轨道之中
AVAsset *srcAsset = [AVAsset assetWithURL:self.videoUrl];
NSArray *trackArray = [srcAsset tracksWithMediaType:AVMediaTypeAudio];
[audioTrack insertTimeRange:audioTimeRange ofTrack:[trackArray objectAtIndex:0] atTime:nextClistartTime error:nil];
// 创建一个输出
AVAssetExportSession *assetExport = [[AVAssetExportSession alloc] initWithAsset:comosition presetName:AVAssetExportPresetMediumQuality];
// 输出类型
assetExport.outputFileType = AVFileTypeQuickTimeMovie;
// 输出地址
assetExport.outputURL = outputFileUrl;
// 优化
assetExport.shouldOptimizeForNetworkUse = YES;
// 合成完毕
[assetExport exportAsynchronouslyWithCompletionHandler:^{
switch ([assetExport status]) {
case AVAssetExportSessionStatusFailed: {
NSLog(@"合成失败：%@",[[assetExport error] description]);
} break;
case AVAssetExportSessionStatusCancelled: {
} break;
case AVAssetExportSessionStatusCompleted: {
NSLog(@"合成成功");
[self saveVideo:outputFileUrl];
dispatch_async(dispatch_get_main_queue(), ^{
[self.progressView setProgress:1.0];
[self.clipView removeFromSuperview];
self.clipView = nil;
[self viewWillAppear:YES];
});
} break;
default: {
break;
} break;
}
}];
}
这样就完成了整个的视频去水印过程，不过处理时间还是有些长，2秒的视频处理了10秒左右...应该是很多优化我没考虑到就单纯实现功能了o(╯□╰)o
源代码git：https://github.com/ahgdwang/WaterMarkDelete.git
https://github.com/ahgdwang/WaterMarkDelete.git
https://github.com/ahgdwang/WaterMarkDelete.git