ceremony.netlify.app

1. Unable To Load Native-hadoop Library For Your Platform Mac Os 2
2. Unable To Load Native-hadoop Library For Your Platform Mac Os 2017
3. Unable To Load Native-hadoop Library For Your Platform Mac Os 7

Tags

hadoop 2.4.1, java 1.7, mac os, native libraries

If you are reading this blog, I assume that you already have Hadoop(v 2.4.1) installed on your OS X machine and that you are bit annoyed by the following error message

Running a hadoop client on Mac OS X and connect to a Kerberized cluster poses some extra challenges. I suggest to use brew, the Mac package manager to conveniently install the Hadoop package: $ brew search hadoop $ brew install hadoop This will install the latest (apache) Hadoop distro, (2.7.3 at th. INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform. Using builtin-java classes where applicable. From Googling this error, I learned that Hadoop Native Libraries are not supported in Mac OS. Is there a way to load the Native Hadoop Library. May 23, 2017  Installation of Hadoop2.x on MacOs(10.12.x) Sierra. This presentation contains all the necessary steps to install and configure Hadoop 2.x distribution on MacOs(10.12.x) Sierra. It can be also be referenced for installing hadoop on OSX(10.11.X) EL Captain. Building Hadoop native library from github and setting it up got covered as well.

Thanks very much for this. Some additions after I spent some time today compiling native libraries for v2.7.0:-zlib is part of osx now, so the command “brew install zlib” may not work, and may in. WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform using builtin-java classes where applicable with hadoop-2.6.0 1 Hadoop 2.6.1 Warning: WARN util.NativeCodeLoader. In case of Spark2 you can enable the DEBUG logging as by invoking the 'sc.setLogLevel('DEBUG')' as following: $ export SPARKMAJORVERSION=2 $ spark-shell -master yarn -deploy-mode client SPARKMAJORVERSION is set to 2, using Spark2 Setting default log level to 'WARN'.

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

If you are only planning to use Hadoop on OS X for development procedures, this error should not bother you. For me it was the same case, but I was just annoyed with this warning message and wanted to try building Native libraries from the source code.

Steps to build Native Hadoop libraries

1. Download source from GitHub

Unable To Load Native-hadoop Library For Your Platform Mac Os 2

• git clone git@github.com:apache/hadoop.git
• git checkout branch-2.4.1

1. Dependencies
   Install cmake and zlib using homebrew package manager

• brew install cmake
• brew install zlib

1. Run maven command

• mvn package -Pdist,native -DskipTests -Dtar

On linux machines, the above procedure should be enough, but not for MAC OS X with Java 1.7. So for that you have to go with few more changes.

If your Mac isn't compatible with macOS Mojave, the installer will let you know. Mojave mac os. Your Mac also needs at least 2GB of memory and 12.5GB of, or up to 18.5GB of storage space when upgrading from OS X Yosemite or earlier.introduced in 2015 or laterintroduced in 2012 or laterintroduced in 2012 or laterintroduced in 2012 or laterintroduced in 2012 or lateriMac Pro (all models)introduced in 2013, plus mid-2010 or mid-2012 models with a.To find your Mac model, memory, storage space, and macOS version, choose About This Mac from the Apple menu . You can upgrade to macOS Mojave from on any of the following Mac models.

Issues faced on Building Native libraries On Mac OS X

1.Missing tools.jar

If you are building using Java 1.7, you would see an error talking about missing tools.jar, which is a bug in Maven JSPC. The related Jira issue is HADOOP-9350. The JSPC Plugin expects classes.jar in ./Classes folder, so we create a symlink.

error:

Exception in thread “main” java.lang.AssertionError: Missing tools.jar at: /Library/Java/JavaVirtualMachines/jdk1.7.0_17.jdk/Contents
/Home/Classes/classes.jar. Expression: file.exists()

Solution: Create a symbolic link to trick Java into believing that classes.jar is same as tools.jar

• sudo mkdir /usr/libexec/java_home/Classes
• sudo ln -s /usr/libexec/java_home/lib/tools.jar /usr/libexec/java_home/Classes/classes.jar

2. Incompatible source code

Some code in Hadoop v2.4.1 is not compatible with Mac system, so need to apply the patch HADOOP-9648.v2.patch and the related Jira issue is HADOOP-10699

error:

Solution: Download the Patch from Jira issue HADOOP-10699. Download link-> HADOOP-9648.v2.patch

• git apply HADOOP-9648.v2.patch
• mvn package -Pdist,native -DskipTests -Dtar

Unable To Load Native-hadoop Library For Your Platform Mac Os 2017

Result

hadoop-dist/target/hadoop-2.4.1/lib/native folder should contain the native libraries. Copy them to hadoop-2.4.1/lib/native folder and restart Hadoop cluster.

Unable To Load Native-hadoop Library For Your Platform Mac Os 7

References

1. Native Libraries Guide documentation page.