如何单元测试一个类是可序列化的火花？

问题描述：

我刚刚在spark中发现了一个类的序列化错误。如何单元测试一个类是可序列化的火花？

=>现在，我想做一个单元测试，但我不知道如何？

注：

故障在已经广播的（解）序列化对象追加。
我要考什么火花会做，断言它会工作一旦部署
类序列化是一个标准类（不区分类别）延伸串行

答

展望火花广播码，我找到了一个方法。但它使用专用的火花代码，因此如果火花内部发生更改，它可能会失效。但它仍然有效。

在封装起始添加一个测试类由org.apache.spark，如：

package org.apache.spark.my_company_tests 

// [imports] 

/** 
* test data that need to be broadcast in spark (using kryo) 
*/ 
class BroadcastSerializationTests extends FlatSpec with Matchers { 

    it should "serialize a transient val, which should be lazy" in { 

    val data = new MyClass(42) // data to test 
    val conf = new SparkConf() 


    // Serialization 
    // code found in TorrentBroadcast.(un)blockifyObject that is used by TorrentBroadcastFactory 
    val blockSize = 4 * 1024 * 1024 // 4Mb 
    val out = new ChunkedByteBufferOutputStream(blockSize, ByteBuffer.allocate) 
    val ser = new KryoSerializer(conf).newInstance() // Here I test using KryoSerializer, you can use JavaSerializer too 
    val serOut = ser.serializeStream(out) 

    Utils.tryWithSafeFinally { serOut.writeObject(data) } { serOut.close() } 

    // Deserialization 
    val blocks = out.toChunkedByteBuffer.getChunks() 
    val in = new SequenceInputStream(blocks.iterator.map(new ByteBufferInputStream(_)).asJavaEnumeration) 
    val serIn = ser.deserializeStream(in) 

    val data2 = Utils.tryWithSafeFinally { serIn.readObject[MyClass]() } { serIn.close() } 

    // run test on data2 
    data2.yo shouldBe data.yo 
    } 
} 

class MyClass(i: Int) extends Serializable { 
    @transient val yo = 1 to i // add lazy to make the test pass: not lazy transient val are not recomputed after deserialization 
}

如何单元测试一个类是可序列化的火花？

相关推荐