Flume and Sqoop for Ingesting Big Data

About course

• Use Flume and Sqoop to import data to HDFS, HBase and Hive from a variety of sources, including Twitter and MySQL

  Import data : Flume and Sqoop play a special role in the Hadoop ecosystem. They transport data from sources like local file systems, HTTP, MySQL and Twitter which hold/produce data to data stores like HDFS, HBase and Hive. Both tools come with built-in functionality and abstract away users from the complexity of transporting data between these systems. 

  Flume: Flume Agents can transport data produced by a streaming application to data stores like HDFS and HBase. 

  Sqoop: Use Sqoop to bulk import data from traditional RDBMS to Hadoop storage architectures like HDFS or Hive. 

  What is Covered:

  Practical implementations for a variety of sources and data stores ..

  • Sources : Twitter, MySQL, Spooling Directory, HTTP
  • Sinks : HDFS, HBase, Hive

  Flume features : 

  Flume Agents, Flume Events, Event bucketing, Channel selectors, Interceptors

  Sqoop features : 

  Sqoop import from MySQL, Incremental imports using Sqoop Jobs

   

Topics covered in this course

• Section 1: You, This Course and Us
  	Lecture 1	You, This Course and Us	01:46
  Section 2: Why do we need Flume and Sqoop?
  	Lecture 2	Why do we need Flume and Sqoop?	18:23
  Section 3: Flume
  	Lecture 3	Installing Flume	02:43
  	Lecture 4	Flume Agent - the basic unit of Flume	10:57
  	Lecture 5	Example 1 : Spool to Logger	14:34
  	Lecture 6	Flume Events are how data is transported	06:07
  	Lecture 7	Example 2 : Spool to HDFS	09:08
  	Lecture 8	Example 3: HTTP to HDFS	09:24
  	Lecture 9	Example 4: HTTP to HDFS with Event Bucketing	05:40
  	Lecture 10	Example 5: Spool to HBase	06:22
  	Lecture 11	Example 6: Using multiple sinks and Channel selectors	09:43
  	Lecture 12	Example 7: Twitter Source with Interceptors	10:48
  Section 4: Sqoop
  	Lecture 13	Installing Sqoop	04:25
  	Lecture 14	Example 8: Sqoop Import from MySQL to HDFS	07:49
  	Lecture 15	Example 9: Sqoop Import from MySQL to Hive	04:26
  	Lecture 16	Example 10: Incremental Imports using Sqoop Jobs	05:24

What you get from this course?

• • Use Flume to ingest data to HDFS and HBase
  • Use Sqoop to import data from MySQL to HDFS and Hive
  • Ingest data from a variety of sources including HTTP, Twitter and MySQL

Who should buy this course?

• • Knowledge of HDFS is a prerequisite for the course
  • HBase and Hive examples assume basic understanding of HBase and Hive shells
  • HDFS is required to run most of the examples, so you'll need to have a working installation of HDFS
  • Engineers building an application with HDFS/HBase/Hive as the data store
  • Engineers who want to port data from legacy data stores to HDFS