从二进制文件中读取巨型int数组

Question

任务

我有一个包含整数并且想在C＃中读取它们的大文件（大约20 GB）。

简单方法

读文件到存储器（成字节数组）是相当快的（使用SSD，整个文件装配到存储器）。但是当我用二进制阅读器（通过内存流）读取这些字节时，ReadInt32-方法比读取文件到内存花费的时间要长得多。我期望成为磁盘IO的瓶颈，但这是转换！

想法和问题

有没有办法直接投全字节数组转换成一个int数组没有给它一个接一个与ReadInt32法转换？写在5499ms

文件

class Program 
{ 
    static int size = 256 * 1024 * 1024; 
    static string filename = @"E:\testfile"; 

    static void Main(string[] args) 
    { 
     Write(filename, size); 
     int[] result = Read(filename, size); 
     Console.WriteLine(result.Length); 
    } 

    static void Write(string filename, int size) 
    { 
     Stopwatch stopwatch = new Stopwatch(); 
     stopwatch.Start(); 
     BinaryWriter bw = new BinaryWriter(new FileStream(filename, FileMode.Create), Encoding.UTF8); 
     for (int i = 0; i < size; i++) 
     { 
      bw.Write(i); 
     } 
     bw.Close(); 
     stopwatch.Stop(); 
     Console.WriteLine(String.Format("File written in {0}ms", stopwatch.ElapsedMilliseconds)); 
    } 

    static int[] Read(string filename, int size) 
    { 
     Stopwatch stopwatch = new Stopwatch(); 
     stopwatch.Start(); 
     byte[] buffer = File.ReadAllBytes(filename); 
     BinaryReader br = new BinaryReader(new MemoryStream(buffer), Encoding.UTF8); 
     stopwatch.Stop(); 
     Console.WriteLine(String.Format("File read into memory in {0}ms", stopwatch.ElapsedMilliseconds)); 
     stopwatch.Reset(); 
     stopwatch.Start(); 

     int[] result = new int[size]; 

     for (int i = 0; i < size; i++) 
     { 
      result[i] = br.ReadInt32(); 
     } 
     br.Close(); 
     stopwatch.Stop(); 
     Console.WriteLine(String.Format("Byte-array casted to int-array in {0}ms", stopwatch.ElapsedMilliseconds)); 

     return result; 
    } 
} 

• 文件在3382ms

Answer 1

读入内存铸造为int数组455ms

字节数组你可以分配一个临时byte[]缓冲方便大小并使用Buffer.BlockCopy方法递增地将字节复制到int[]阵列。

BinaryReader reader = ...; 
int[] hugeIntArray = ...; 

const int TempBufferSize = 4 * 1024 * 1024; 
byte[] tempBuffer = reader.ReadBytes(TempBufferSize); 
Buffer.BlockCopy(tempBuffer, 0, hugeIntArray, offset, TempBufferSize); 

凡offset是电流（当前迭代）目的地hugeIntArray阵列中的起始索引。