Hadoop

Resources: https://github.com/wavestone-cdt/hadoop-attack-library

Terminology

Cluster - Refers to all the systems that together make the datalake.

Node - A single host or computer in the Hadoop cluster.

NameNode - A node that is responsible for keeping the directory tree of the Hadoop file system.

DataNode - A slave node that stores files according to the instructions of a NameNode.

Primary NameNode - The current active node responsible for keeping the directory structure.

Secondary NameNode - The backup node which will perform a seamless takeover of the directory structure should the Primary NameNode become unresponsive. There can be more than one Secondary NameNode in a cluster, but only one Primary active at any given time.

Master Node - Any node that is executing a Hadoop "management" application such as HDFS Manager or YARN Resource Manager.

Slave Node - Any node that runs a Hadoop "worker" application such as HDFS or MapReduce. It should be noted that a single node can be both a Master and Slave node at the same time.

Edge Node - Any node that is hosting a Hadoop "user" application such as Zeppelin or Hue. These are applications that users can use to perform processing on the data stored in the datalake.

Kerberised - The term given for a datalake that has security enabled through Kerberos.

Apache Hadoop Services

HDFS - Hadoop Distributed File System is the primary storage application for unstructured data such as files

Hive - Hive is the primary storage application for structured data. Think of it as a massive database.

YARN - Main resource manager application of Hadoop, used to schedule jobs in the cluster

MapReduce - Application executor of Hadoop to process vast amounts of data. It consists of a Map procedure, which performs filtering and sorting, and a reduce method, which performs a summary operation.

HUE - A user application that provides a GUI for HDFS and Hive.

Zookeeper - Provides operational services for the cluster to set the configuration of the cluster in question.

Spark - Engine for large-scale data processing.

Kafka - A message broker to build pipelines for real-time data processing.

Ranger - Used for the configuration of privilege access control over the resources in the datalake.

Zeppelin - A web-based notebook application for interactive data analytics.