Data Pipeline & Cleansing

Our technology choice for the data pipeline is Luigi, because of its simple code level dependency mapping eliminating the need for complex configuration files. There may be a need for an intermediate data store for the pipeline to use while it processes the data. The data pipeline orchestrates the tasks involved in cleansing and transforming the data.