11 Hadoop

Software platform to easily process vast amounts of data. The main features are: Scalable : It can reliably store and process petabytes. Economical : It distributes the data and processing across…

HDFS Architecture NameNode Manages File System Namespace Maps a file name to a set of blocks Maps a block to the DataNodes where it resides Cluster Configuration Management Replication Engine for…

Shell Commands There are two types of shell commands: User Commands hdfs dfs – runs filesystem commands on the HDFS hdfs fsck – runs a HDFS filesystem checking command Administration Commands hdfs…

YARN Splits up the two major functions of JobTracker Global Resource Manager - Cluster resource management Application Master - Job scheduling and monitoring (one per application). The Application…

Introduction MapReduce is a programming model and an associated implementation for processing and generating big data sets with a parallel, distributed algorithm on a cluster. A MapReduce program is…

Input Splitter Is responsible for splitting your input into multiple chunks (default is 64MB). These chunks are then used as input for your mappers Splits on logical boundaries. Typically, you can…

By default, Hadoop uses FIFO to schedule jobs. Alternate scheduler options: capacity and fair Capacity Scheduler Jobs are submitted to queues Jobs can be prioritized Queues are allocated a fraction…