AWS ETL pipelines

Examples of architecture using AWS ETL and DB services:

• Data Pipelines (S3 to S3 and S3 to RDS)
• Data Pipeline (S3 to Redshift)
• S3 to DynamoDB
• Glue Example 1
• Glue Example 2
• Glue Example 3
• Kinesis (Streams and Firehose)
• Step-functions
• EventBridge

All source code can be found in this repository and the scripts for the various examples are stored in the respective named folders.

Dependencies

For running some of the scripts locally, first install poetry

pip install poetry

and then install the dependencies from the poetry.lock file Ref

poetry install

Installing dependencies from lock file

The command below will spawn a new shell with a virtual env containing the newly installed dependencies

$ poetry shell

Spawning shell within /Users/rk1103/Library/Caches/pypoetry/virtualenvs/aws-etl-fV9WWBi4-py3.9
(base) rk1103@Ryans-MacBook-Air aws_etl % . /Users/rk1103/Library/Caches/pypoetry/virtualenvs/aws-etl-fV9WWBi4-py3.9/bin/activate
(aws-etl-fV9WWBi4-py3.9) (base) rk1103@Ryans-MacBook-Air aws_etl %

Most of the resources are created via cloudformation templates. The process for doing this is described here