Create S3-Athena Dataset

info

• Follow the steps mentioned below.
• Create three s3-athena datasets for Charges, Transactions and Denials.
• These datasets will be used to load the output data from the ETL job in the next steps.
• Total time taken for this task: 5 Minutes.
• Pre-requisites: Setup Amorphic and login and Create a Domain

Tidbits

• Add your ⚠️userid⚠️ as a suffix to all the resources that you create.
• This will avoid contention with other users as Amorphic maintains a global metadata catalog. 💡
• This is mandatory as multiple users are following these workshop instructions.

Create S3-Athena Dataset

• Click on Catalog --> Datasets on the left side navigation-bar.
• Click on a ➕ New Dataset icon at the top right corner.
• Enter all the required details as mentioned below and click on Next

Dataset details

"Dataset Name" : "HL_charges_<user_id>"
"Description" : "Dataset to store Charges data"
"Domain" : "hlworkshop"
"Connection Type" : "api"
"File Type" : "parquet"
"Target Location" : "S3-Athena"
"Update Method" : "Append"
"Data Validation" : "Disable"
"Data Files Have Headers" : "Yes"
"Custom Delimiter" : " , "
"Skip LZ(Validation) Process" : "No"
"SerDe" : "Parquet Hive SerDe"
"Enable Malware Detection" : "No"
"Enable Data Profiling" : "No"
"Enable Data Cleanup" : "No"
"Enable Metrics Collection" : "No"
"Enable Life Cycle Policy" : "No"

Review & Submit

• Review the dataset details and click on Create

Note

For s3-athena type datasets, Dataset registration will be in pending state until the schema is added to the dataset

Add Schema to the Dataset

• Click on Click Here button on the Registration Pending box
• Select Upload Schema Definition JSON File option
• Upload the charges schema JSON file by clicking on click here button (OR) drag and drop the charges schema JSON file into the upload box.
• Click on Continue button.
• Review and confirm the schema.
• Click on Publish Dataset to finish dataset registration for Charges dataset.

Note

Simillarly follow the above mentioned steps to create Transactions and Denials datasets

Transactions Dataset

Denials Dataset