Amazon Redshift Copy Activity

Introduction

An Amazon Redshift Copy activity reads data from an Amazon S3 bucket, inserts records into a table at Amazon Redshift, and is intended to be used as a target to consume data in an operation. After configuring an Amazon Redshift connection, you can configure as many Amazon Redshift activities as you like for each Amazon Redshift connection.

Creating an Amazon Redshift Activity

From the design canvas, open the Connectivity tab of the design component palette:

Use the Show dropdown to filter on Endpoints, and then click the Amazon Redshift connection block to display activities that are available to be used with an Amazon Redshift connection:

To create an activity that can be configured, drag the activity block from the palette to the operation.

For more information about the parts of an operation and adding activities to operations, see Operation Creation and Configuration.

Configuring an Amazon Redshift Copy Activity

Follow these steps to configure an Amazon Redshift Copy activity:

• Step 1: Enter a Name and Select a Schema
• Step 2: Select a Table
• Step 3: Specify the Load Approach
• Step 4: Review the Data Schemas

Step 1: Enter a Name and Select a Schema

• Name: Enter a name to use to identify the Amazon Redshift Copy activity. The name must be unique for each Amazon Redshift Copy activity and must not contain forward slashes (/) or colons (:).

• Select a Schema: This section displays schemas available in the Amazon Redshift endpoint. When reopening an existing activity configuration, only the selected schema is displayed instead of reloading the entire schema list.

  • Selected Schema Name: After a schema is selected, it is listed here.

  • Search: Enter any part of the schema name into the search box to filter the list of schemas. The search is not case-sensitive. If schemas are already displayed within the table, the table results are filtered in real time with each keystroke. To reload schemas from the endpoint when searching, enter search criteria and then refresh, as described below.

  • Refresh: Click the refresh icon or the word Refresh to reload schemas from the Amazon Redshift endpoint. This may be useful if you have recently added schemas to Amazon Redshift. This action refreshes all metadata used to build the table of schemas displayed in the configuration.

  • Selecting a Schema: Within the table, click anywhere on a row to select a schema. Only one schema can be selected. The information available for each schema is fetched from the Amazon Redshift endpoint:

    • Schema: The schema from Amazon Redshift.

  TIP: If the table does not populate with available schemas, the Amazon Redshift connection may not be successful. Ensure you are connected by reopening the connection and retesting the credentials.

• Save & Exit: If enabled, click to save the configuration for this step and close the activity configuration.

• Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Step 2: Select a Table

• Select a Table: This section displays tables available in the Amazon Redshift endpoint. When reopening an existing activity configuration, only the selected table is displayed instead of reloading the entire table list.

  • Selected Schema Name: The schema name selected in the previous step is listed here.

  • Select Table Name: After a table is selected, it is listed here.

  • Search: Enter any part of the table name into the search box to filter the list of tables. The search is not case-sensitive. If tables are already displayed within the table, the table results are filtered in real time with each keystroke. To reload tables from the endpoint when searching, enter search criteria and then refresh, as described below.

  • Refresh: Click the refresh icon or the word Refresh to reload tables from the Amazon Redshift endpoint. This may be useful if you have recently added tables to Amazon Redshift. This action refreshes all metadata used to build the table of tables displayed in the configuration.

  • Selecting a Table: Within the table, click anywhere on a row to select a table. Only one table can be selected. The information available for each table is fetched from the Amazon Redshift endpoint:

    • Table Name: The table name from Amazon Redshift.

    • Schema: The table schema from Amazon Redshift.

    • Catalog: The table catalog from Amazon Redshift.

  TIP: If the table does not populate with available tables, the Amazon Redshift connection may not be successful. Ensure you are connected by reopening the connection and retesting the credentials.

• Back: Click to temporarily store the configuration for this step and return to the previous step.

• Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Step 3: Specify the Load Approach

• Approach: Specify the credentials approach to be used for the AWS bucket, one of Role-based or Key-based:

  • Role-based: If using the Role-based approach to access the AWS bucket:

    • Credentials IAM Role: Enter the IAM role associated with the credentials.

  • Key-based: If using the Key-based approach to access the AWS bucket:

    • AWS Access Key ID: Enter the AWS access key ID.
    • AWS Secret Access Key: Enter the AWS secret access key.

• From: Enter the Amazon bucket that the data is to be copied from.

• Region: Enter the Amazon region of the bucket that the data is to be copied from.

• Delimiter: Enter the delimiter that is to be used when copying the data.

• Back: Click to temporarily store the configuration for this step and return to the previous step.

• Next: Click to temporarily store the configuration for this step and continue to the next step. The configuration will not be saved until you click the Finished button on the last step.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Step 4: Review the Data Schemas

• Data Schema: The request and response data schemas for Amazon Redshift are displayed. If the operation uses a transformation, the data schemas are displayed again later during the transformation mapping process, where you can map to target fields using source objects, scripts, variables, custom values, and more.

  The Amazon Redshift connector uses the Amazon Redshift JDBC Driver and Amazon Redshift SQL Commands. Refer to the Amazon Redshift documentation and the Amazon Redshift System Overview documentation for additional information.

• Refresh: Click the refresh icon or the word Refresh to regenerate schemas from the endpoint. This action also regenerates the schema in other locations throughout the project where the same schema is referenced, such as in an adjacent transformation.

• Back: Click to temporarily store the configuration for this step and return to the previous step.

• Finished: Click to save the configuration for all steps and close the activity configuration.

• Discard Changes: After making changes, click to close the configuration without saving changes made to any step. A message asks you to confirm that you want to discard changes.

Next Steps

After configuring an Amazon Redshift Copy activity, complete the configuration of the operation by adding and configuring other activities, transformations, or scripts as operation steps. You can also configure an operation's operation settings, which include the ability to chain operations together that are in the same or different workflows.

After an Amazon Redshift activity has been created, menu actions for that activity are accessible from the project pane in either the Workflows or the Components tabs, and from the design canvas. See Activity Actions Menu for details.

Amazon Redshift Copy activities can be used as a target with these operation patterns:

• Transformation Pattern
• Two-Transformation Pattern (as the first or second target)

Other patterns are not valid using Amazon Redshift Copy activities. See the validation patterns on the Operation Validity page.

A typical use case is to use an Amazon Redshift Copy activity in the Two-Transformation Pattern. In this example, the first transformation (Amazon Redshift Copy Request) creates a request structure that is passed to the Amazon Redshift Copy activity. The second transformation (Amazon Redshift Copy Response) receives the response structure, which is then written to a variable by a Variable Write activity (Write Amazon Redshift Copy Response) and a message is then logged by the Write to Operation Log script:

To use the activity with scripting functions, write the data to a temporary location and then use that temporary location in the scripting function.

When ready, deploy and run the operation and validate behavior by checking the operation logs.