七月算法深度学习 第三期 学习笔记-第六节 图像风格转换
Example:
Google Deep Dream: https://deepdreamgenerator.com/
Prisma: https://prisma-ai.com/
NeuralStyle: A Neural Algorithm of Artistic Style by Leon A. Gatys, Alexander S.Ecker, and Matthias Bethge,2015
DeepArt: https://deepart.io/
术语说明：
风格图片 S （style image）
内容图片 C （content image）
合成图片 X （style-transferred image）
阅读资料：A Neural Algorithm of Artistic Style (Leon A. Gatys, Alexander S. Ecker, Matthias Bethge): https://arxiv.org/abs/1508.06576
图片风格迁移的大致流程：
1.导入需要的一堆库：
import time
from PIL import Image
import numpy as np
from keras import backend
from keras.models import Model
from keras.applications.vgg16 import VGG16
from scipy.optimize import fmin_l_bfgs_b
from scipy.misc import imsave
2.读入图片
读取内容和风格两个图片，定义图片的长宽、路径、并设成统一大小：
height = 512
width = 512
#读入内容图片
content_image_path = 'images/isabell.jpg'
content_image = Image.open(content_image_path)
content_image = content_image.resize((height, width))
content_image
#读入风格图片
style_image_path = 'images/styles/wave.jpg'
style_image = Image.open(style_image_path)
style_image = style_image.resize((height, width))
style_image
通过增加一层Dimension将图片数据变为四维，为了方便将两张图片连接起来
content_array = np.asarray(content_image, dtype='float32')
content_array = np.expand_dims(content_array, axis=0)
print(content_array.shape)
style_array = np.asarray(style_image, dtype='float32')
style_array = np.expand_dims(style_array, axis=0)
print(style_array.shape)
(1, 512, 512, 3)
(1, 512, 512, 3)
我们需要把数据拟合进VGG-16中，来使用一套CNN模型:https://arxiv.org/abs/1409.1556
1）保证色调色域的一致：VGG16是在ImageNet上跑出来的，我们也把我们的数据『平均化』到ImageNet的色域上，也就是，把我们的图片统一剪掉一个ImageNet的平均值，使得我们自己的图片色彩分布更符合ImageNet的原始图片库:http://image-net.org/
2）保证色彩维度位置一致：VGG16里不是RGB而是BGR，我们把三个维度调换一下。
content_array[:, :, :, 0] -= 103.939
content_array[:, :, :, 1] -= 116.779
content_array[:, :, :, 2] -= 123.68
content_array = content_array[:, :, :, ::-1]
style_array[:, :, :, 0] -= 103.939
style_array[:, :, :, 1] -= 116.779
style_array[:, :, :, 2] -= 123.68
style_array = style_array[:, :, :, ::-1]
在使用Keras的backend之前，我们把合成照片的backend variable给搞好，应该是跟我们的原图一样大小,再把三个图片都concate在一起
content_image = backend.variable(content_array)
style_image = backend.variable(style_array)
combination_image = backend.placeholder((1, height, width, 3))
input_tensor = backend.concatenate([content_image,
style_image,
combination_image], axis=0)
3. 调用pretrain的VGG16
它会自动下载fchollet的VGG16的weights
model = VGG16(input_tensor=input_tensor, weights='imagenet',
include_top=False)
此刻，我们可以看一看整个神经网络的长相
layers = dict([(layer.name, layer.output) for layer in model.layers])
layers
4.优化
这里用三个损失方程来优化我们的图片：content loss、style loss、total variation loss
附上权值：
content_weight = 0.025
style_weight = 5.0
total_variation_weight = 1.0
初始化loss为0：
loss = backend.variable(0.)
The content loss
接下来我们处理content loss
def content_loss(content, combination):
return backend.sum(backend.square(combination - content))
layer_features = layers['block2_conv2']
content_image_features = layer_features[0, :, :, :]
combination_features = layer_features[2, :, :, :]
loss += content_weight * content_loss(content_image_features,
combination_features)
The style loss
以及style loss
先写出Gram Matrix的方法
def gram_matrix(x):
features = backend.batch_flatten(backend.permute_dimensions(x, (2, 0, 1)))
gram = backend.dot(features, backend.transpose(features))
return gram
The total variation loss
以及整体loss
def total_variation_loss(x):
a = backend.square(x[:, :height-1, :width-1, :] - x[:, 1:, :width-1, :])
b = backend.square(x[:, :height-1, :width-1, :] - x[:, :height-1, 1:, :])
return backend.sum(backend.pow(a + b, 1.25))
loss += total_variation_weight * total_variation_loss(combination_image)
5.用Gradients解决Optimisation问题
一个是loss，一个是我们的合成图片
grads = backend.gradients(loss, combination_image)
细化准备一个我们的Evaluator的方法
记录下每次的loss和新的gradient值
outputs = [loss]
outputs += grads
f_outputs = backend.function([combination_image], outputs)
def eval_loss_and_grads(x):
x = x.reshape((1, height, width, 3))
outs = f_outputs([x])
loss_value = outs[0]
grad_values = outs[1].flatten().astype('float64')
return loss_value, grad_values
class Evaluator(object):
def __init__(self):
self.loss_value = None
self.grads_values = None
def loss(self, x):
assert self.loss_value is None
loss_value, grad_values = eval_loss_and_grads(x)
self.loss_value = loss_value
self.grad_values = grad_values
return self.loss_value
def grads(self, x):
assert self.loss_value is not None
grad_values = np.copy(self.grad_values)
self.loss_value = None
self.grad_values = None
return grad_values
evaluator = Evaluator()
在这里用一种比较快的Quasi-Newton算法来寻找最优解：https://en.wikipedia.org/wiki/Limited-memory_BFGS
给个10轮
x = np.random.uniform(0, 255, (1, height, width, 3)) - 128.
iterations = 10
for i in range(iterations):
print('Start of iteration', i)
start_time = time.time()
x, min_val, info = fmin_l_bfgs_b(evaluator.loss, x.flatten(),
fprime=evaluator.grads, maxfun=20)
print('Current loss value:', min_val)
end_time = time.time()
print('Iteration %d completed in %ds' % (i, end_time - start_time))
然后，得到了我们最新的图像的数字表达形式，反过去，把数据变回成图片
x = x.reshape((height, width, 3))
x = x[:, :, ::-1]
x[:, :, 0] += 103.939
x[:, :, 1] += 116.779
x[:, :, 2] += 123.68
x = np.clip(x, 0, 255).astype('uint8')
Image.fromarray(x)
图片输入前
最终输出
总结：