ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[682112,156]

运行出错结果如下:

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[682112,156]


出错原因主要和自己的算法有关,在图像识别的程序中,我的filters的数量分配如下:

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[682112,156]

Filters的数量越多,同一张图片经过这些filters提取特征后占用显卡内存数量越大,经过这两层众多数量的filter卷积后,占用的内存极大。另外:

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[682112,156]

图片经过这些filters提取特征后缓存在显卡内存中,经过flaten展平,在经过一层包含1024个神经元的神经网络学习,每一批(batch)数据经过整个算法处理的内容都缓存在显卡中,可想而知内存占用非常大。最后还和batchsize有关,显卡是按块(batch)一批一批进入显卡训练,因此batch越大,占用内存越大。经过调整后的算法如下:

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[682112,156]

把filter的数量,batchsize以及dense中的神经元数量改小后算法就能跑了,运行如下:

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[682112,156]