尧图网络 高端网站定制 · 原创设计
免费咨询热线
400-888-6620
免费获取方案
【运维与安装】Hadoop 3.3.1 :Setting up a Single Node Cluster.
最近公司搞国产化项目里用到的hadoop 2.10.1版本 不支持国产化hadoop 3.1.1 版本支持ARM架构的国产CPU。hadoop的版本修改势必会影响项目里对API、restfulAPI、flink with hadoop相关使用所以本地先搭建一个hadoop 3.1.1 测试一下版本升级对项目的影响。参考官网搭建一个本地的hadoop官网Hadoop: Setting up a Single Node Cluster.文章目录1. Required Software2. Download3. Prepare to Start the Hadoop Cluster4. HDFS on a Single Node4.1 configurationa. etc/hadoop/core-site.xml:b. etc/hadoop/hdfs-site.xml:Setup passphraseless ssh (省略)4.2. Executiona. Format the filesystem:b. Start NameNode daemon and DataNode daemon:c. Browse the web interface for the NameNode; by default it is available at:d. When you’re done, stop the daemons with:5. YARN on a Single Node5.1. Configure parameters as follows:5.2. Start ResourceManager daemon and NodeManager daemon:5.3. Browse the web interface for the ResourceManager; by default it is available at:5.4. When you’re done, stop the daemons with:1. Required Softwarejava 和 sshJava™ must be installed. Recommended Java versions are described at HadoopJavaVersionsssh must be installed and sshd must be running to use the Hadoop scripts thatmanage remote Hadoop daemonsif the optional start and stop scripts are to be used.Additionally, it is recommmended that pdsh also be installed for better ssh resource management.2. DownloadTo get a Hadoop distribution, download a recent stable release from one of the Apache Download Mirrors.3. Prepare to Start the Hadoop Cluster解压并在 hadoop-env.sh 设置java home。Unpack the downloaded Hadoop distribution. In the distribution, edit the file etc/hadoop/hadoop-env.sh to define some parameters as follows:#setto the root of your Java installationexportJAVA_HOME/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home尝试命令$ bin/hadoopThis will display the usage documentation for the hadoop script.Now you are ready to start your Hadoop cluster in one of the three supported modes:我们这次关注单节点的安装Pseudo-Distributed Operation (伪集群模式)Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.4. HDFS on a Single Node4.1 configurationa. etc/hadoop/core-site.xml:要配置一个数据目录用于存储数据否则可能hadoop默认选择的目录没有权限使用。org.apache.hadoop.hdfs.server.common.InconsistentFSStateException:Directory/private/tmp/hadoop-lianggao/dfs/name is in an inconsistent state:storage directory doesnotexistorisnotaccessible.at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:392)at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:243)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1197)at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:779)at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:677)at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:764)at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:1018)at org.apache.hadoop.hdfs.server.namenode.NameNode.init(NameNode.java:991)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1767)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1832)configurationpropertynamefs.defaultFS/namevaluehdfs://localhost:9000/value/propertypropertynamehadoop.tmp.dir/namevalue/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/value/property/configurationb. etc/hadoop/hdfs-site.xml:configurationpropertynamedfs.replication/namevalue1/value/property/configurationSetup passphraseless ssh (省略)如需要参考官网官网4.2. Executiona. Format the filesystem:$ bin/hdfs namenode-format看到这些信息格式化成功2022-08-1217:43:11,039INFO common.Storage:Storage directory/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/dfs/name has been successfully formatted.2022-08-1217:43:11,069INFO namenode.FSImageFormatProtobuf:Saving image file/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/dfs/name/current/fsimage.ckpt_0000000000000000000usingno compression2022-08-1217:43:11,200INFO namenode.FSImageFormatProtobuf:Image file/Users/lianggao/MyWorkSpace/002install/hadoop-3.3.1/hadoop_repo/dfs/name/current/fsimage.ckpt_0000000000000000000 of size403bytes saved in0seconds.如果格式化失败需要先删除nn的管理目录因为格式化的时候是创建了nn文件的管理目录 common.Storage: Storage directory /data/hadoopdata/name has been successfully formatted.b. Start NameNode daemon and DataNode daemon:$ sbin/start-dfs.shc. Browse the web interface for the NameNode; by default it is available at:NameNode - http://localhost:9870/d. When you’re done, stop the daemons with:$ sbin/stop-dfs.sh5. YARN on a Single Node5.1. Configure parameters as follows:etc/hadoop/mapred-site.xml:configurationpropertynamemapreduce.framework.name/namevalueyarn/value/propertypropertynamemapreduce.application.classpath/namevalue$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*/value/property/configurationetc/hadoop/yarn-site.xml:configurationpropertynameyarn.nodemanager.aux-services/namevaluemapreduce_shuffle/value/propertypropertynameyarn.nodemanager.env-whitelist/namevalueJAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME/value/property/configuration5.2. Start ResourceManager daemon and NodeManager daemon:$ sbin/start-yarn.sh5.3. Browse the web interface for the ResourceManager; by default it is available at:ResourceManager - http://localhost:8088/5.4. When you’re done, stop the daemons with:$ sbin/stop-yarn.sh
RELATED

相关推荐

工业SPI隔离实战:用数字隔离器MAX14431与PIC24HJ设计稳定通信链路

工业SPI隔离实战:用数字隔离器MAX14431与PIC24HJ设计稳定通信链路

/* MD / 富文本中的 .toc(含博客园搬家等嵌套结构);.toc-box 在侧栏,不受影响 */#content_views .toc,/* 编辑器常在目录前后插入空 p(:empty 仍占 20px),一并去掉避免顶空隙 */#content_views.markdown_views > p:empty:has(+ .toc),#content_views.markdown_views …

📅 2026/11/3 18:10:36
基于MAX14431BAEE+与PIC18LF45K40的工业级隔离SPI通信链路设计与实现

基于MAX14431BAEE+与PIC18LF45K40的工业级隔离SPI通信链路设计与实现

/* MD / 富文本中的 .toc(含博客园搬家等嵌套结构);.toc-box 在侧栏,不受影响 */#content_views .toc,/* 编辑器常在目录前后插入空 p(:empty 仍占 20px),一并去掉避免顶空隙 */#content_views.markdown_views > p:empty:has(+ .toc),#content_views.markdown_views …

📅 2026/11/3 18:10:36
S7-1500与WinCC GroupDisplay组显示配置实战:多工位总览画面搭建与调试

S7-1500与WinCC GroupDisplay组显示配置实战:多工位总览画面搭建与调试

/* MD / 富文本中的 .toc(含博客园搬家等嵌套结构);.toc-box 在侧栏,不受影响 */#content_views .toc,/* 编辑器常在目录前后插入空 p(:empty 仍占 20px),一并去掉避免顶空隙 */#content_views.markdown_views > p:empty:has(+ .toc),#content_views.markdown_views …

📅 2026/11/3 18:10:36
MORE NEWS

更多资讯

📰

如何抽取实体间的关系?GLiFormer Large v1 联合关系抽取实战教程

如何抽取实体间的关系?GLiFormer Large v1 联合关系抽取实战教程 【免费下载链接】gliformer-large-v1 项目地址: https://ai.gitcode.com/hf_mirrors/knowledgator/gliformer-large-v1 GLiFormer Large v1 是面向信息抽取的多任务大模型,支持联…

📰

OpenWhispr 语音转文字听写指南:如何用全局热键把说话直接变成文字

OpenWhispr 语音转文字听写指南:如何用全局热键把说话直接变成文字 【免费下载链接】openwhispr Voice-to-text dictation app with local (Nvidia Parakeet/Whisper) and cloud models (BYOK). Privacy-first and available cross-platform. 项目地址: https://g…

📰

26个数据集、91%的F1:GLiFormer Large v1基准评测数据全解读

人工智能NLP大模型 【免费下载链接】gliformer-large-v1 项目地址: https://ai.gitcode.com/hf_mirrors/knowledgator/gliformer-large-v1 点击查看 免费下载 GLiFormer Large v1 是一个 5.756 亿参数的开源多任务信息抽取模型:一个 DeBERTa 编码器同时…

📰

BiliBiliToolPro 部署教程:从本地一键命令到零维护云函数,5 种方式完整指南

BiliBiliToolPro 部署教程:从本地一键命令到零维护云函数,5 种方式完整指南 【免费下载链接】BiliBiliToolPro B 站(bilibili)自动任务工具,支持docker、青龙、k8s等多种部署方式。全面拥抱AI。敏感肌也能用。 项目地…

📰

Whisky:免费在Mac运行Windows程序的完整指南

Whisky:免费在Mac运行Windows程序的完整指南 【免费下载链接】Whisky A modern Wine wrapper for macOS built with SwiftUI 项目地址: https://gitcode.com/gh_mirrors/wh/Whisky Whisky 是一个用 SwiftUI 写的 macOS 原生 Wine 封装器,让你不碰…

📰

《经典机器学习实战》第 1 课:导论——是什么、简史、负责任 AI 与标准流程

本文收录于专栏 《经典机器学习实战》 摘要:从规则引擎的破产到机器学习的定义,一趟走完 AI 七十年简史、负责任 AI 六原则与七步标准流程。你将看到 13 维特征让手写规则彻底失效而逻辑回归拿到 98.1%,以及一个整体准确率 85% 的模型如何对少…

TODAY

今日更新

THIS WEEK

本周精选

THIS MONTH

本月热门

读完文章,想聊聊您的网站?

告诉我们您的行业与需求,资深顾问一对一梳理方案与报价,全程免费。

📞 💬