windows环境运行datax缺少本地hadoop环境报错:Could not locate executable null\bin\winutils.exe
目录
一、完整报错
十二月 07, 2023 10:01:05 上午 org.apache.hadoop.util.Shell getWinUtilsPath
严重: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:356)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:371)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:364)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2807)
at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2802)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2668)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170)
at com.alibaba.datax.plugin.writer.hdfswriter.HdfsHelper.getFileSystem(HdfsHelper.java:74)
at com.alibaba.datax.plugin.writer.hdfswriter.HdfsWriter$Job.init(HdfsWriter.java:49)
at com.alibaba.datax.core.job.JobContainer.initJobWriter(JobContainer.java:704)
at com.alibaba.datax.core.job.JobContainer.init(JobContainer.java:304)
at com.alibaba.datax.core.job.JobContainer.start(JobContainer.java:113)
at com.alibaba.datax.core.Engine.start(Engine.java:94)
at com.alibaba.datax.core.Engine.entry(Engine.java:176)
at com.alibaba.datax.core.Engine.main(Engine.java:246)
十二月 07, 2023 10:01:05 上午 org.apache.hadoop.util.NativeCodeLoader <clinit>
警告: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
二、解决
2.1、下载本地hadoop环境
地址(免费下载):windows本地hadoop环境包
2.2、解压并配置环境变量
环境变量配置如下:
然后重启pycharm或电脑再运行datax.py即可
三、临时解决方案
利用Python的os模块中datax.py文件上面加上如下代码,加入临时环境变量也可。
os.environ['HADOOP_HOME'] = r"D:\Hadoop"
四、其他
如发现运行datax后,日志有乱码情况,可以在终端或控制台输入以下命令来解决。
chcp 65001