【日常踩坑】Spark踩坑合集

Spark 2.4.X 无法读取 Hive 数据

场景分析: Hive 部分数据表存在数据, Spark 2.1.1 可以读取, 但是 Spark 2.4.X 无法读取

分析过程:

1. 针对可读取数据表和不可读取数据表进行分析

hadoop fs -ls -R can_read_table_path | awk '{print $8}' | awk -F/ '{printf " |-";for(i=3;i<NF;i++){printf "--"} print $NF}'
 |---------------000000_0
 |---------------000001_0
 |---------------000002_0
hadoop fs -ls -R cannot_read_table_path | awk '{print $8}' | awk -F/ '{printf " |-";for(i=3;i<NF;i++){printf "--"} print $NF}'
 |-------------HIVE_UNION_SUBDIR_1
 |---------------000000_0
 |---------------000001_0
 |---------------000002_0
 |-------------HIVE_UNION_SUBDIR_2
 |---------------000000_0
 |---------------000001_0
 |---------------000002_0

此时可以看出两者的文件目录结构不同, 估计该部分是两个数据在不同版本无法读取的主要原因, 即高版本 Spark 并未设置递归读

2. 针对 Spark 2.4.X 进行参数设置(Google)

• How to pick up all data into hive from subdirectories
• Hive on tez的insert union 子目录的问题

通过上述资料, 可以知道需要设置如下参数:

SET hive.mapred.supports.subdirectories=true;
SET mapred.input.dir.recursive=true;
SET mapreduce.input.fileinputformat.input.dir.recursive=true;

但是不管用, 针对 Hive 是起作用的(因为系统本身 Hive 已经设置如上参数啦~).

3. 分析 Spark 2.1.1 到 Spark 2.4.X 之间针对 Hive 部分的改造

分析原因在于两个 Spark 基本上共用同一份 Hive 和 Spark 设置, 并处于同一 Yarn 平台上, 两者 DIFF 情况应当在于 Spark 高版本的某些优化导致的

通过对更新日志和Spark JIRA搜索, 定位问题可能是由于spark.sql.hive.convertMetastoreOrc参数默认值的变化

• Spark SQL Upgrading Guide
• SPARK-28098

In version 2.3 and earlier, Spark converts Parquet Hive tables by default but ignores table properties like . This happens for ORC Hive table properties like in case of , too. Since Spark 2.4, Spark respects Parquet/ORC specific table properties while converting Parquet/ORC Hive tables. As an example, would generate Snappy parquet files during insertion in Spark 2.3, and in Spark 2.4, the result would be uncompressed parquet files.TBLPROPERTIES (parquet.compression 'NONE')TBLPROPERTIES (orc.compress 'NONE')spark.sql.hive.convertMetastoreOrc=trueCREATE TABLE t(id int) STORED AS PARQUET TBLPROPERTIES (parquet.compression 'NONE')

Since Spark 2.0, Spark converts Parquet Hive tables by default for better performance. Since Spark 2.4, Spark converts ORC Hive tables by default, too. It means Spark uses its own ORC support by default instead of Hive SerDe. As an example, would be handled with Hive SerDe in Spark 2.3, and in Spark 2.4, it would be converted into Spark’s ORC data source table and ORC vectorization would be applied. To set to restores the previous behavior.CREATE TABLE t(id int) STORED AS ORCfalsespark.sql.hive.convertMetastoreOrc

更新记录含义: Spark 2.4 以后, Spark将采用自身的 SerDe 而非采用 Hive 的SerDe 处理 Hive 的 ORC 数据表(SerDe是Serialize/Deserilize的简称,目的是用于序列化和反序列化)

4. 定位 Spark 源码

根源代码在于: HiveStrategies.scala中 RelationConversions 类, 可以看到根据配置不同, 对于 ORC 文件采用 org.apache.spark.sql.execution.datasources.orc.OrcFileFormat 或 org.apache.spark.sql.hive.orc.OrcFileFormat 处理方式, 其中采用 org.apache.spark.sql.execution.datasources.orc.OrcFileFormat 会导致如上问题. 并未对比两个的区别, 有兴趣的同学可以查看一下~

case class RelationConversions(
    conf: SQLConf,
    sessionCatalog: HiveSessionCatalog) extends Rule[LogicalPlan] {
  private def convert(relation: HiveTableRelation): LogicalRelation = {
    val serde = relation.tableMeta.storage.serde.getOrElse("").toLowerCase(Locale.ROOT)

    // Consider table and storage properties. For properties existing in both sides, storage
    // properties will supersede table properties.
    if (serde.contains("parquet")) {
      val options = relation.tableMeta.properties.filterKeys(isParquetProperty) ++
        relation.tableMeta.storage.properties + (ParquetOptions.MERGE_SCHEMA ->
        conf.getConf(HiveUtils.CONVERT_METASTORE_PARQUET_WITH_SCHEMA_MERGING).toString)
      sessionCatalog.metastoreCatalog
        .convertToLogicalRelation(relation, options, classOf[ParquetFileFormat], "parquet")
    } else {
      val options = relation.tableMeta.properties.filterKeys(isOrcProperty) ++
        relation.tableMeta.storage.properties
      if (conf.getConf(SQLConf.ORC_IMPLEMENTATION) == "native") {
        sessionCatalog.metastoreCatalog.convertToLogicalRelation(
          relation,
          options,
          classOf[org.apache.spark.sql.execution.datasources.orc.OrcFileFormat],
          "orc")
      } else {
        sessionCatalog.metastoreCatalog.convertToLogicalRelation(
          relation,
          options,
          classOf[org.apache.spark.sql.hive.orc.OrcFileFormat],
          "orc")
      }
    }
  }
}

总结: 在 Spark 2.3 以后, Spark 增加 spark.sql.hive.convertMetastoreOrc 参数, 设定 ORC 文件使用采用向量化 Reader, 但是会引申由于 Spark SQL 无法与 Hive SQL 采用同一 SerDe 从而对 Parquet/Hive 数据表产生处理上的不同, 因此需要限定如下参数从而让 Spark 使用 Hive 的SerDe.

SET spark.sql.hive.convertMetastoreOrc=false;
SET spark.sql.hive.convertMetastoreParquet=false;

pyspark 报错 GLIBC NOT FOUND

原因: 因集群部分节点系统版本较低,所以编译Python的机器中,glibc版本需要低于 2.14(可以使用ls -l /lib64/libc.so.6或者ldd --version查看),否则executor无法启动Python进程,会报错类似错误

Error from python worker:
  pythondir/python-3.7.10/bin/python3: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by pythondir/python-3.7.10/bin/python3)
  pythondir/python-3.7.10/bin/python3: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by pythondir/python-3.7.10/bin/python3)

编译 Python

python 3.7 需要进行 openssl 编译, 而 python 3.6 则不需要, 如下步骤以 python 3.7 举例

# 利用 docker 构建 GLIBC 环境
docker run --name "build_python" -it --rm --cpus=4 centos:centos6 /bin/bash
# 如下为 docker 内部运行命令
## 替换源: 清华源等均已下架 Centos 6
sed -e 's|^mirrorlist=|#mirrorlist=|g' \
    -e 's|^#baseurl=http://mirror.centos.org|baseurl=https://vault.centos.org|g' \
    -i.bak \
    /etc/yum.repos.d/CentOS-*.repo
## 安装必备包
yum -y install wget perl gcc gcc-c++ zip unzip
yum install -y ncurses-libs zlib-devel mysql-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel libffi-devel
## 下载openssl-1.0.2u版本并编译
cd ~ &&
wget https://www.openssl.org/source/old/1.0.2/openssl-1.0.2u.tar.gz &&
tar zxvf openssl-1.0.2u.tar.gz && 
cd openssl-1.0.2u &&
./config --prefix=/output/openssl-1.0.2u --openssldir=/output/openssl-1.0.2u/openssl &&
make -j && make install
## 下载Python3.7 并配置
cd ~ &&
wget https://npm.taobao.org/mirrors/python/3.7.10/Python-3.7.10.tgz &&
tar zxvf Python-3.7.10.tgz && 
cd Python-3.7.10 &&
./configure --enable-optimizations --prefix=/output/python-3.7.10
## 修改编译参数
vi Modules/Setup
## 大约在221行左右,修改SSL路径,并且将这四行的注释去掉
SSL=/output/openssl-1.0.2u
_ssl _ssl.c \
   -DUSE_SSL -I$(SSL)/include -I$(SSL)/include/openssl \
   -L$(SSL)/lib -lssl -lcrypto
## 编译&安装
make -j && make install
## 安装所需包
cd /output/python-3.7.10
./bin/pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
./bin/pip3 install YOUR_NEED_BAG
## 压缩Python
zip -r python37.zip ./
## 获取libffi
cd /usr/lib64 &&
zip -r libs.zip ./libffi.so.* &&
mv libs.zip /output/
# 导出
docker cp build_python:/output/python-3.7.10/python37.zip ~/Downloads
docker cp build_python:/output/libs.zip ~/Downloads
# 推送到hdfs
hadoop fs -put ~/Downloads/python37.zip /YOUR_WORK_PATH/

参考资料