hadoop 源码编译 导入eclipse
hadoop2版本出来后,突然心血来潮想编译下hadoop2的代码并导入eclipse,以备以后有时间了进行代码研究用,现在总结了下编译过程以及如何导入到elipse。一下是必备条件
- hadoop版本:hadoop的尝鲜版release-2.0.3-alpha,svn地址:http://svn.apache.org/repos/asf/hadoop/common/tags
- JDK1.6
-
ANT 及ivy,下载ant,配置环境变量,将ivy的下载jar包放到ant的lib目录下,即:%ANT_HOME%/lib
- mvn3.0
- 装protoc buffer http://protobuf.googlecode.com/files/protobuf-2.4.1.tar.gz,将下载的的包解压,然后将protoc.exe复制到 cygwin的bin目录(%cygwin_home%/bin),
- 安装cygwin,在win下面,首先下载cygwin:http://www.cygwin.com/ 在path中设置路径
以下是需要注意修改的地方:
1、由于编译的源码过大,以至于mvn编译过程中会出现内存溢出,所以需要设置mvn的编译内存,打开mvn 的bin目录下的mvn.bat文件,添加set MAVEN_OPTS= -Xms128m -Xmx1024m。
2、由于在执行过程中mvn的mojo不能执行sh脚本文件,如在<executable>saveVersion.sh</executable>
,所以需要做如下修改,在elease-2.0.3-alpha\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-common目录下和pom.xml同一目录,新建saveVersion.bat的文件,文件内容是调用saveVersion.sh文件,如:sh scripts\saveVersion.sh %1 %2
3、修改release-2.0.3-alpha\hadoop-yarn-project\hadoop-yarn\hadoop-yarn-common\scripts\saveVersion.sh文件,修改user=`who 为user=hdfs,因为在windows环境下活不到用户名,会出现编译page-info的类出错
最后按着hadoop的wiki(http://wiki.apache.org/hadoop/EclipseEnvironment)进行编译打包:
$ mvn install -DskipTests $ mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs=true
Note: This may take a while the first time, as all libraries are fetched from the internet, and the whole build is performed.
In Eclipse
After the above, do the following to finally have projects in Eclipse ready and waiting for you to go on that scratch-itching development spree:
For Common
-
File -> Import...
- Choose "Existing Projects into Workspace"
- Select the hadoop-common-project directory as the root directory
- Select the hadoop-annotations, hadoop-auth, hadoop-auth-examples and hadoop-common projects
- Click "Finish"
-
File -> Import...
- Choose "Existing Projects into Workspace"
- Select the hadoop-assemblies directory as the root directory
- Select the hadoop-assemblies project
- Click "Finish"
- To get the projects to build cleanly:
- * Add target/generated-test-sources/java as a source directory for hadoop-common
-
* You may have to add then remove the JRE System Library to avoid errors due to access restrictions
For HDFS
-
File -> Import...
- Choose "Existing Projects into Workspace"
- Select the hadoop-hdfs-project directory as the root directory
- Select the hadoop-hdfs project
- Click "Finish"
For MapReduce
-
File -> Import...
- Choose "Existing Projects into Workspace"
- Select the hadoop-mapreduce-project directory as the root directory
- Select the hadoop-mapreduce-project project
- Click "Finish"
相关推荐
Hadoop源码编译好的源码(eclipse可直接导入)直接eclipse->import->Existing Maven Projects即可无错误阅读Hadoop源码。亲给个好评吧~
hadoop cdh 编译所需下载的所有软件,很全的
Hadoop源码编译
hadoop源码进行编译,需要预安装的一些软件包,包含protobuf,snappy,findbugs,hadoop源码包,jdk1.7,maven等。
包含hadoop2.6.0源码和eclipse开发hadoop所用插件,下载即可用。
hadoop源码编译所需要的maven仓库,省去编译时下载时间。
里面是 protobuf-2.5.0.tar.gz apache-ant-1.9.9-bin.tar.gz 和meavn
Hadoop源码编译.pdf
如何把hadoop源码关联到eclipse工程中
Hadoop2.7.1源码(可直接导入Eclipse)
提供了安装hadoop源码安装步骤、文档、安装包,只需要按照步骤做,就能完成。注意上面所有安装步骤都必须实现,及maven远程仓库配置。
hadoop2x-eclipse-plugin
win 7 64上编译 Hadoop 2.7.3 源码 的真实经历。
该文档详细介绍了,在windows下,对hadoop源码进行编译的全过程。 内用截图,更直观!
Hadoop源码编译,我用的方法是创建maven工程,使用maven编译后每个模块对应生成target目录,为调试做准备
5.1 前期准备工作 5.2 jar 包安装 5.3 编译源码 5.4 常见的问题及解决方案 2.7.0 版本的问题汇总帖子 http://www.tuicoo
Ubuntu16.04+Eclipse neon.1+maven3.3.9成功导入的Hadoop2.7.3全部源码工程文件,已解决了全部的报错
hadoop环境搭建和eclipse开发
WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 需要进行hadoop源码的编译