Spark-ml模型保存为PMML格式，PMML version不兼容问题。

前言

最近使用spark-ml做数据模型训练，考虑到怎么把模型部署到线上环境。
本尝试spring-boot启动spark-local模式预测，发现线上预测需要200ms~300ms，耗时太多。
所以考虑把spark模型转换为pmml格式。
我的spark集群版本为2.1.3

---

本机local模式

采用了JPMML-SparkML，本地环境配置如下：

<dependency>
	<groupId>org.apache.spark</groupId>
	<artifactId>spark-mllib_2.11</artifactId>
	<version>${spark.version}</version>
	<scope>provided</scope>
	<exclusions>
		<exclusion>
			<groupId>org.jpmml</groupId>
			<artifactId>pmml-model</artifactId>
		</exclusion>
	</exclusions>
</dependency>
<dependency>
	<groupId>org.jpmml</groupId>
	<artifactId>jpmml-sparkml</artifactId>
	<version>1.2.13</version>
</dependency>

local模式保存为pmml代码：

import org.dmg.pmml.PMML;
import org.jpmml.model.JAXBUtil;
import org.jpmml.sparkml.PMMLBuilder;

...

		StructType schema = trainData.schema();
        PMML pmml = new PMMLBuilder(schema, pipelineModel).build();

        String targetFile = "D://model/pmml/pipemodel";
        FileOutputStream fis = new FileOutputStream(targetFile);
        try {
            JAXBUtil.marshalPMML(pmml, new StreamResult(fis));
        } catch (JAXBException e) {
            e.printStackTrace();
        }
...

---

但是放在集群上无法使用。
集群上spark-mllib_2.11是包含pmml-model的，没权限动整个公司的spark，无法exclude，默认为4.2版本。不会主动加载jpmml-sparkml的4.3版本 pmml-model

集群方式

所以采用shaded方式重新打包，重启起一个项目。

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.tc.ml</groupId>
    <artifactId>pmml-model</artifactId>
    <version>1.0</version>

    <dependencies>
        <dependency>
            <groupId>org.jpmml</groupId>
            <artifactId>jpmml-sparkml</artifactId>
            <version>1.2.13</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>3.1.0</version>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                        <configuration>
                            <relocations>
                                <relocation>
                                    <pattern>org.dmg.pmml</pattern>
                                    <shadedPattern>org.shaded.dmg.pmml</shadedPattern>
                                </relocation>
                                <relocation>
                                    <pattern>org.jpmml.model</pattern>
                                    <shadedPattern>org.shaded.jpmml.model</shadedPattern>
                                </relocation>
                            </relocations>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

注意下github上jpmml-sparkml shade打包方式为：

	<relocations>
	  <relocation>
		<pattern>org.dmg.pmml</pattern>
		<shadedPattern>org.shaded.dmg.pmml</shadedPattern>
	 </relocation>
	 <relocation>
		 <pattern>org.jpmml</pattern>
		 <shadedPattern>org.shaded.jpmml</shadedPattern>
	  </relocation>
  </relocations>

但无法使用VectorAssembler，
所以第二个relocation改为

<relocation>
  <pattern>org.jpmml.model</pattern>
  <shadedPattern>org.shaded.jpmml.model</shadedPattern>
</relocation>

建依赖改为打包后的pmml-model-1.0.jar包

 <dependency>
     <groupId>org.jpmml</groupId>
      <artifactId>jpmml-sparkml</artifactId>
      <version>1.2.13-shaded</version>
      <scope>system</scope>
     <systemPath>${project.basedir}/lib/pmml-model-1.0.jar</systemPath>
</dependency>

保存为hdfs

import org.jpmml.sparkml.PMMLBuilder;
import org.shaded.dmg.pmml.PMML;
import org.shaded.jpmml.model.JAXBUtil;
....
代码省略
....
        StructType schema = trainData.schema();
        PMML pmml = new PMMLBuilder(schema, pipelineModel).build();

        String targetFile = "/data/twms/traffichuixing/model/pmml/pipepmml";
        Path path=new Path(targetFile);
        FSDataOutputStream fos = fileSystem.create(path);
        try {
            JAXBUtil.marshalPMML(pmml, new StreamResult(fos));
        } catch (JAXBException e) {
            e.printStackTrace();
        }finally {
            fos.close();
        }
.....
代码省略
....