DL4J hello world
背景：之前尝试TensorFlow训练保存pb模型给到spark用，感觉还是性能太慢了；开始寻求在spark上跑深度学习的方法，权衡sparkNet和DL4J后选择。
参考官网 https://deeplearning4j.org/cn/quickstart 先弄了个例子：
步骤1：克隆到本地
F:\spark project\dl4j-examples>git clone https://github.com/deeplearning4j/dl4j-examples.git
Cloning into 'dl4j-examples'...
remote: Enumerating objects: 201, done.
remote: Counting objects: 100% (201/201), done.
remote: Compressing objects: 100% (133/133), done.
error: RPC failed; curl 56 OpenSSL SSL_read: SSfL_ERROR_SYSCALL, errno 10054
atal: the remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
上面问题参考：https://stackoverflow.com/questions/21277806/fatal-early-eof-fatal-index-pack-failed 如下解决：
F:\spark project\dl4j-examples>
F:\spark project\dl4j-examples>git config --global core.compression 0
F:\spark project\dl4j-examples>git clone --depth 1 https://github.com/deeplearning4j/dl4j-examples.git
Cloning into 'dl4j-examples'...
remote: Enumerating objects: 768, done.
remote: Counting objects: 100% (768/768), done.
remote: Compressing objects: 100% (547/547), done.
remote: Total 768 (delta 161), reused 491 (delta 97), pack-reused 0 eceiving objects: 99% (761/768), 22.93 MiB | 76.00 KiB/s
Receiving objects: 100% (768/768), 22.94 MiB | 165.00 KiB/s, done.
Resolving deltas: 100% (161/161), done.
感觉执行到这里可以了，毕竟例子都下下来了，然后我又往下执行了几行：
F:\spark project\dl4j-examples>git fetch --unshallow
fatal: not a git repository (or any of the parent directories): .git
F:\spark project\dl4j-examples>git fetch --depth=2147483647
fatal: not a git repository (or any of the parent directories): .git
F:\spark project\dl4j-examples>git init
Initialized empty Git repository in F:/spark project/dl4j-examples/.git/
F:\spark project\dl4j-examples>git fetch --unshallow
fatal: --unshallow on a complete repository does not make sense
F:\spark project\dl4j-examples>git fetch --depth=2147483647
F:\spark project\dl4j-examples>git pull --all
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.
git pull <remote> <branch>
If you wish to set tracking information for this branch you can do so with:
git branch --set-upstream-to=<remote>/<branch> master
步骤2：大概要半小时，然后需要5G多磁盘空间
mvn clean install
这里我没有单独安装maven，而是偷懒在idea的其他项目中通过Execute Maven Goal在指定路径F:\spark project\dl4j-examples\dl4j-examples 下执行mvn clean install的
步骤3：挑个例子跑起来：
如org.deeplearning4j.examples.feedforward.classification.MLPClassifierLinear，
IDEA运行测试用例报错如下：
Error running 'AuthServer': Command line is too long. Shorten command line for AuthServeror also for Application default configuration.
解决办法：
修改项目下 .idea\workspace.xml，找到标签 <component name="PropertiesComponent"> ， 在标签里加一行 <property name="dynamic.classpath" value="true" />
执行结果：
步骤4：到自己的spark里面运行
先添加依赖：
<dependency>
<groupId>org.datavec</groupId>
<artifactId>datavec-api</artifactId>
<version>1.0.0-beta5</version>
</dependency>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native-platform</artifactId>
<version>1.0.0-beta5</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-core</artifactId>
<version>1.0.0-beta5</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>dl4j-spark_2.11</artifactId>
<version>1.0.0-beta5</version>
</dependency>
<dependency>
<groupId>com.beust</groupId>
<artifactId>jcommander</artifactId>
<version>1.27</version>
</dependency>
复制org.deeplearning4j.legacyExamples.mlp.MnistMLPExample并稍作修改得到：
import org.apache.spark.SparkConf
import org.apache.spark.api.java.JavaSparkContext
import org.deeplearning4j.datasets.iterator.impl.MnistDataSetIterator
import org.deeplearning4j.eval.Evaluation
import org.deeplearning4j.nn.conf.NeuralNetConfiguration
import org.deeplearning4j.nn.conf.layers.DenseLayer
import org.deeplearning4j.nn.conf.layers.OutputLayer
import org.deeplearning4j.nn.weights.WeightInit
import org.deeplearning4j.spark.impl.multilayer.SparkDl4jMultiLayer
import org.deeplearning4j.spark.impl.paramavg.ParameterAveragingTrainingMaster
import org.nd4j.linalg.activations.Activation
import org.nd4j.linalg.dataset.DataSet
import org.nd4j.linalg.learning.config.Nesterovs
import org.nd4j.linalg.lossfunctions.LossFunctions
import java.util
object MnistMLPExample {
val batchSizePerWorker=16
val numEpochs = 2
def main(args: Array[String]): Unit = {
val sparkConf = new SparkConf
System.setProperty("hadoop.home.dir", "D:\\火狐下载\\hadoop-common-2.2.0-bin-master")
sparkConf.setMaster("local[*]")
sparkConf.setAppName("DL4J Spark MLP Example")
val sc = new JavaSparkContext(sparkConf)
sc.setLogLevel("WARN")
//Load the data into memory then parallelize
//This isn't a good approach in general - but is simple to use for this example
val iterTrain = new MnistDataSetIterator(batchSizePerWorker, true, 12345)
val iterTest = new MnistDataSetIterator(batchSizePerWorker, true, 12345)
val trainDataList = new util.ArrayList[DataSet]
val testDataList = new util.ArrayList[DataSet]
while ( {
iterTrain.hasNext
}) trainDataList.add(iterTrain.next)
while ( {
iterTest.hasNext
}) testDataList.add(iterTest.next)
val trainData = sc.parallelize(trainDataList)
val testData = sc.parallelize(testDataList)
//Create network configuration and conduct network training
val conf = new NeuralNetConfiguration.Builder()
.seed(12345)
.activation(Activation.LEAKYRELU)
.weightInit(WeightInit.XAVIER)
.updater(new Nesterovs(0.1))// To configure: .updater(Nesterovs.builder().momentum(0.9).build())
.l2(1e-4)
.list()
.layer(new DenseLayer.Builder().nIn(28 * 28).nOut(500).build())
.layer(new DenseLayer.Builder().nOut(100).build())
.layer(new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
.activation(Activation.SOFTMAX).nOut(10).build())
.build();
//Configuration for Spark training: see http://deeplearning4j.org/spark for explanation of these configuration options
val tm = new ParameterAveragingTrainingMaster.Builder(batchSizePerWorker) //Each DataSet object: contains (by default) 32 examples
.averagingFrequency(5)
.workerPrefetchNumBatches(2) //Async prefetching: 2 examples per worker
.batchSizePerWorker(batchSizePerWorker)
.build();
//Create the Spark network
val sparkNet = new SparkDl4jMultiLayer(sc, conf, tm)
//Execute training:
var i = 0
while ( {
i < numEpochs
}) {
sparkNet.fit(trainData)
println("Completed Epoch {}", i)
i += 1;
}
//Perform evaluation (distributed)
// Evaluation evaluation = sparkNet.evaluate(testData);
val evaluation = sparkNet.doEvaluation(testData, 64, new Evaluation(10))(0) //Work-around for 0.9.1 bug: see https://deeplearning4j.org/releasenotes
println("***** Evaluation *****")
println(evaluation.stats)
//Delete the temp training files, now that we are done with them
tm.deleteTempFiles(sc)
println("***** Example Complete *****")
}
}
运行结果为：
......
15:03:14,619 INFO ~ Completed Epoch 1
15:03:20,498 INFO ~ ***** Evaluation *****
15:03:20,502 INFO ~
========================Evaluation Metrics========================
# of classes: 10
Accuracy: 0.9608
Precision: 0.9605
Recall: 0.9607
F1 Score: 0.9605
Precision, recall & F1: macro-averaged (equally weighted avg. of 10 classes)
=========================Confusion Matrix=========================
0 1 2 3 4 5 6 7 8 9
---------------------------------------------------
5807 0 10 3 11 17 24 3 40 8 | 0 = 0
1 6583 50 17 9 8 1 7 54 12 | 1 = 1
24 11 5759 24 27 11 19 31 47 5 | 2 = 2
14 16 99 5717 3 113 8 29 96 36 | 3 = 3
6 14 25 2 5615 1 31 5 16 127 | 4 = 4
28 8 14 65 15 5180 39 8 43 21 | 5 = 5
29 8 10 0 21 49 5775 0 26 0 | 6 = 6
11 24 77 13 35 3 2 6006 14 80 | 7 = 7
18 39 24 56 14 45 21 9 5600 25 | 8 = 8
22 12 5 43 115 28 2 71 43 5608 | 9 = 9
Confusion matrix format: Actual (rowClass) predicted as (columnClass) N times
==================================================================
15:03:20,502 INFO ~ Attempting to delete temporary directory: /tmp/hadoop-lenovo/dl4j/1572245786579_-2c362db/0/
15:03:22,346 INFO ~ Deleted temporary directory: /tmp/hadoop-lenovo/dl4j/1572245786579_-2c362db/0/
15:03:22,347 INFO ~ ***** Example Complete *****
DeepLearning4J环境搭建、测试完成，接下来了解下DLJ4的细节...