Hadoop

Dropping a managed table removes its metastore entry and deletes its data files. Dropping an external table removes only the metastore entry and leaves the files untouched. Declare EXTERNAL whenever another system owns the data.

The NameNode keeps the filesystem namespace in memory: directories, files, permissions, and which blocks make up each file. DataNodes store the actual block bytes on local disks. File data never flows through the NameNode.

A Hadoop MapReduce job runs map, then shuffle and sort, then reduce. One map task handles each InputSplit; its output is partitioned and sorted on local disk, fetched by reducers over the network, merged by key, and written out by the OutputFormat.

YARN splits cluster management three ways: the ResourceManager schedules containers across the whole cluster, a NodeManager runs and monitors containers on each machine, and every application gets its own ApplicationMaster that requests containers and drives that job.

The Hive Metastore is a relational catalog mapping table and partition names to storage paths, columns, types, file formats and statistics. Engines such as Trino, Presto and Flink read it so that directories of files behave as SQL tables.

A combiner is a reduce-style function the framework may run zero, one or many times over a mapper's own output. An arithmetic mean is not decomposable that way, so averaging partial groups and then averaging those averages gives a wrong result.

The active NameNode writes edits to a quorum of JournalNodes; the standby tails them and receives DataNode block reports, so it is hot. ZooKeeper plus a ZKFC per NameNode elects the active, and journal epochs plus fencing block the stale writer.

Every file, directory and block is an object held in the NameNode's heap, and the cost is per object, not per byte. Ten million tiny files consume as much master memory as ten million huge ones while storing almost no data.

Each YARN container is granted a fixed memory amount, and the NodeManager monitors the container's whole process tree against it. When total resident memory crosses the grant the container is killed, because YARN protects the node from oversubscription rather than letting the machine swap or OOM.

The Secondary NameNode only merges the edit log into a new fsimage so restarts stay fast — it is not a standby. In an HA cluster the standby NameNode already tails every edit, so it performs the checkpoint and the Secondary NameNode role disappears.

core-site.xml holds cluster-wide settings like fs.defaultFS, hdfs-site.xml configures NameNode and DataNode behaviour, yarn-site.xml the ResourceManager and NodeManagers. Each overrides the matching *-default.xml shipped in the jars, and a job can override again unless the property is marked final.

Choose HBase when you need millisecond random reads and writes of individual rows by key, and updates in place. Choose a Hive table on HDFS when the workload is large sequential scans and aggregations over immutable files.

A large HDFS block amortises disk seek time over a long sequential read and keeps the NameNode's per-byte metadata cost tiny. Blocks are logical, so a 5 MB file consumes 5 MB of disk, not 128 MB.

With the default policy, the first replica goes on the writing client's own node (or a random DataNode if the client is off-cluster), the second on a node in a different rack, and the third on another node in that same second rack.

The client asks the NameNode for a block and a list of DataNodes, then streams packets to the first DataNode, which forwards to the second, which forwards to the third. Acknowledgements travel back up the chain before packets are released.

An InputSplit is the logical byte range one map task will process, plus host hints for locality — not a physical copy of data. With FileInputFormat its size is max(minSize, min(maxSize, blockSize)), so by default it equals the HDFS block size of 128 MB.

Both divide a YARN cluster into hierarchical queues, but the Capacity Scheduler starts from guaranteed percentages per queue with an elastic ceiling, while the Fair Scheduler starts from equal sharing among running applications and pulls resources back toward each queue's fair share over time.

Each writing task creates its own file in every partition it touches, so files multiply as tasks times partitions. Funnel each partition to one writer with DISTRIBUTE BY the partition columns, enable Hive's merge settings, and use coarser partition granularity.

Almost always key skew. The default HashPartitioner sends every value for a given key to one reduce task, and a single key's group cannot be split across reducers, so one hot key pins one task. Speculative execution does not help, because the retry processes the same data.

ACCEPTED means YARN took the application but has not launched its ApplicationMaster container yet. The cause is always missing capacity: the queue is full, its ApplicationMaster budget is exhausted, a per-user limit binds, or no healthy node can host the requested container size.

Size three budgets separately: raw storage as logical data times the replication factor plus headroom for non-DFS use and node rebuilds; NameNode heap from the count of files, directories and blocks rather than bytes; and worker memory, cores and disks from the workload's container profile.

Airflow wins when workflows must be generated and tested as code, span systems beyond the cluster, and be operated by people who need a usable UI. Oozie's XML workflows are static, YARN-bound and awkward to test.

Rank jobs by how much the MapReduce model actually costs them. Multi-job chains and iterative algorithms pay an HDFS round trip between every stage and repay a rewrite; stable single-pass map-only jobs are already IO-bound and rarely justify the risk.

Shape queues around workload classes and SLAs rather than around org charts: give SLA-bound pipelines guaranteed capacity with limited borrowing, let ad-hoc and batch work share a large elastic queue, cap concurrency and per-user limits, and enable preemption only where redoing work is cheap.

Sqoop takes a split column, queries its minimum and maximum, divides that range into equal slices, and gives each mapper a SELECT with its own WHERE range. Each mapper opens a separate connection to the source database.

It becomes a map-only job: no partitioning, no sort, no shuffle. Each map task writes its records straight through the OutputFormat to HDFS as part-m-NNNNN files, and any configured combiner never runs.

The ResourceManager notices the ApplicationMaster is gone and starts a fresh attempt in a new container elsewhere, up to yarn.resourcemanager.am-max-attempts (default 2). When attempts run out the whole application is marked FAILED, and by default its earlier containers are killed too.

With work-preserving restart, containers already running on NodeManagers keep running. The standby ResourceManager becomes active, rebuilds application state from its ZooKeeper state store, and NodeManagers and ApplicationMasters resync with it. Clients retry transparently; new scheduling stalls for the failover window.

Erasure coding suits large, cold, rarely-read files: a Reed-Solomon scheme such as RS-6-3 tolerates three failures at about 1.5x storage overhead instead of replication's 3x. It costs read locality, CPU on recovery, and does not support append.