MetalPipe: Modules for ETL Pipelines

MetalPipe is a set of modules for building configuration-driven, efficient ETL pipelines with a minimum amount of custom code.

Contents:

• Overview
  • Why is it?
  • What is it?
  • What isn’t it?
  • MetalPipe pipelines
  • Rolling your own MetalNode class
  • Composing and configuring MetalNode objects
• Quickstart
  • Install MetalPipe
  • Write a configuration file
  • Run the pipeline
• Configuration
• Node and Data Lifecycle
  • The Node Lifecycle
  • The data journey
• Monitoring
  • Logging table
• Treehorn
  • Using Treehorn
  • Summing up
• API documentation
  • Node module
  • Civis-specific node types
  • Data structures module
  • Network nodes module
  • MetalPipeMessage module
  • Trigger module
  • Batch module
  • MetalPipeQueue module
• License

Indices and tables

• Index
• Module Index
• Search Page