ByteHouse 云数仓版 Spark Connector 连接器专门用于通过 Spark 将数据加载到 ByteHouse 云数仓版。本文将介绍通过 Spark SQL，以及 EMR 支持的 Servless Spark 两种方式连接 ByteHouse 并处理数据。

• Java 8
• Scala 2.12 及以上版本
• Spark 3.3 或 3.5

请按照下面的方法，在程序中配置以下驱动的依赖项。

1. ByteHouse Spark Connector。
2. ClickHouse JDBC 驱动程序。
3. ByteHouse JDBC 驱动程序。

Maven 依赖

对于要使用 Spark connector 连接器进行编译的 Maven 项目，请将以下依赖项添加到项目的 pom.xml 文件中。

1. 添加如下依赖。

   Spark 版本	依赖
   Spark 3.3	<dependency> <groupId>com.clickhouse</groupId> <artifactId>clickhouse-jdbc</artifactId> <version>0.4.6</version> </dependency> <dependency> <groupId>com.bytedance.bytehouse</groupId> <artifactId>driver-java</artifactId> <version>1.1.58</version> <classifier>all</classifier> </dependency> <dependency> <groupId>com.bytedance.bytehouse</groupId> <artifactId>clickhouse-spark-runtime-3.3_2.12</artifactId> <version>0.8.0.6</version> </dependency>
   Spark 3.5	<dependency> <groupId>com.clickhouse</groupId> <artifactId>clickhouse-jdbc</artifactId> <version>0.4.6</version> </dependency> <dependency> <groupId>com.bytedance.bytehouse</groupId> <artifactId>driver-java</artifactId> <version>1.1.58</version> <classifier>all</classifier> </dependency> <dependency> <groupId>com.bytedance.bytehouse</groupId> <artifactId>clickhouse-spark-runtime-3.5_2.12</artifactId> <version>0.8.0.10</version> </dependency>

2. 将以下存储库添加到 pom.xml 文件：

   <repository>
       <id>bytedance</id>
       <name>ByteDance Public Repository</name>
       <url>https://artifact.bytedance.com/repository/releases</url>
   </repository>
   

Spark SQL

您可以参考下面的命令，基于 Spark SQL CLI 连接到 ByteHouse。

export SPARK_LOCAL_IP=localhost
export SPARK_HOME=/opt/tiger/spark
export CLICKHOUSE_HOST=bytehouse-cn-beijing.volces.com
export CLICKHOUSE_PASSWORD=<your-api-key>
export CLICKHOUSE_DATABASE=<database>
export CLICKHOUSE_VW=<your-virtual-warehouse>

$SPARK_HOME/bin/spark-sql \
  --conf spark.sql.catalog.clickhouse=com.bytehouse.ByteHouseCatalog \
  --conf spark.sql.catalog.clickhouse.host=${CLICKHOUSE_HOST:-127.0.0.1} \
  --conf spark.sql.catalog.clickhouse.password=${CLICKHOUSE_PASSWORD:-} \
  --conf spark.sql.catalog.clickhouse.database=${CLICKHOUSE_DATABASE} \
  --conf spark.sql.catalog.clickhouse.option.vw=${CLICKHOUSE_VW} \
  --conf spark.driver.bindAddress=127.0.0.1 \
  --conf spark.bytehouse.write.batchSize=50000 \
  --conf spark.bytehouse.write.format=jdbc \
  --jars clickhouse-jdbc-0.4.6-all.jar,driver-java-1.1.58.jar,clickhouse-spark-runtime-3.5_2.12-0.8.0.10.jar

Servless Spark

Servless Spark 方式适用于火山引擎 EMR（E-MapReduce）服务。您可以参考下面的命令，基于 Servless Spark 方式连接到 ByteHouse。

set serverless.spark.analysis=true;
set spark.hadoop.fs.tos.skip.resolve = true;
set las.cross.vpc.access.enabled = true;
set las.cross.vpc.vpc.id = <your-vpc-id>;
-- 使用可用区 B 子网
set las.cross.vpc.subnet.id = <your-subnet-id>;
set las.cross.vpc.security.group.id = <your-securitygroup-id>;
set las.cross.vpc.accountId=<your-account-id>;
set las.cluster.type = vke;
set emr.serverless.spark.only.parse.enabled  = true;
set serverless.spark.access.key=<your-tos-ak>;
set serverless.spark.secret.key=<your-tos-sk>;
set spark.sql.catalog.clickhouse=com.bytehouse.ByteHouseCatalog;
-- 私网连接地址
set spark.sql.catalog.clickhouse.host=tenant-<your-account-id>-cn-beijing.bytehouse.ivolces.com;
-- 公网连接地址
-- set spark.sql.catalog.clickhouse.host=bytehouse-cn-beijing.volces.com;
set spark.sql.catalog.clickhouse.protocol=http;
set spark.sql.catalog.clickhouse.password=<your-api-key>;
set spark.sql.catalog.clickhouse.database=<database>;
set spark.sql.catalog.clickhouse.option.vw=<your-virtual-warehouse>;

set las.spark.jar.depend.jars = [{"schema":"","fileName":"tos://test-bh-tos/cdw/driver-1.1.58-all.jar"},{"schema":"","fileName":"tos://test-bh-tos/cdw/clickhouse-jdbc-0.4.6-all.jar"},{"schema":"","fileName":"tos://test-bh-tos/cdw/clickhouse-spark-runtime-3.5_2.12-0.8.0.10.jar"}];

use clickhouse;

-- 查询 hive
-- select * from spark_catalog.hive_db.hive_table;
-- select * from people_t_copy;
insert into people_t_copy select * from people_t; 

配置参数