RDI on Redis Cloud quick start

Learn how to create a data pipeline between a PostgreSQL source database created with Terraform and a Redis Cloud target database.

Redis Cloud

The rdi-cloud-automation GitHub repository contains a Terraform script that quickly sets up a PostgreSQL source database on an EC2 instance and all required permissions and network setup to connect it to a Redis Cloud target database.

Note:
This guide is for demonstration purposes only. It is not recommended for production use.

Prerequisites

To follow this guide, you need to:

  1. Create a Redis Cloud Pro database hosted on Amazon Web Services (AWS).

    Turn on Multi-AZ replication and manually select the availability zones when creating the database.

  2. Install the AWS CLI and set up credentials for the CLI.

  3. Install Terraform.

Create a data integration workspace

Before you can create your first Data Integration pipeline for a Redis Cloud subscription, you must first deploy the cloud infrastructure needed to host the pipeline and run the workers associated with the pipeline. In Redis Cloud, this is called a Workspace. See Create and manage Data Integration workspace for more information.

To create a Data Integration workspace for an existing Pro subscription:

  1. From the Redis Cloud console, select Data Integration from the left-hand menu. If you don't have any workspaces yet, select Create workspace to go to the Create workspace page.

    The create workspace button.

    If you already have a workspace deployed, you'll see your current workspaces. Select New workspace to go to the Create workspace page.

    The new workspace button.

    You can also go to the Data Integration tab from your subscription or database page and select Create workspace to go to the Create workspace page for your subscription.

    The create workspace button.

  2. Select your Pro subscription from the list if it's not already selected.

    The select pro subscription drop down.

  3. A Data Integration subnet (CIDR) is automatically generated for you. Each RDI workspace uses a dedicated /22 CIDR.

    For AWS, the RDI workspace CIDR must:

    • Be in the same RFC 1918 private address range as the subscription VPC's primary CIDR: 10.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16.
    • Not overlap with existing subscription, peering, transit gateway (TGW), application, database, or RDI workspace CIDR ranges.

    For example, if the subscription VPC's primary CIDR is 10.238.252.0/24, then 192.168.0.0/22 is invalid because it is in a different RFC 1918 range. An unused range such as 10.239.0.0/22 is valid.

    If the automatic suggestion is missing or unsuitable, select another unused /22 CIDR in the same private range. For more information, see VPC CIDR block association restrictions.

    The select pro subscription drop down.

  4. Select Create workspace to create your workspace.

    The create workspace button.

Your workspace will be created in the background. You can select Create pipeline to create your pipeline while the workspace is provisioning, or you can select Create pipeline later to go back to the Redis Cloud console.

Get required ARNs

This example creates one PostgreSQL source. You can add more sources after the pipeline is running.

  1. On the Redis Cloud console, open your target database's Data Integration tab and select Add pipeline.

    The Add pipeline control is available while the workspace is being created.

  2. In Settings, select your target database and choose Hash or JSON as the default data structure, then select Continue.

  3. In Add sources, select PostgreSQL.

  4. Enter a Source name of your choice, for example inventory-postgres.

  5. Select Continue to open Configure source.

  6. Under Source connectivity, copy the Role ARN. Use it as redis_privatelink_arn in the Terraform configuration.

    The source connectivity Role ARN and availability zones.

  7. Under Secrets, copy the Role ARN. Use it as redis_secrets_arn in the Terraform configuration.

    The Role ARN in the Secrets section.

  8. Select Save & exit while you create the source resources.

Create the source database and network resources

  1. Clone or download the rdi-cloud-automation GitHub repository.

  2. In a terminal window, go to the examples/aws-ec2-privatelink directory.

  3. Run terraform init to initialize the Terraform working directory.

  4. Open the example.tfvars file and edit the following variables:

    • region: The AWS region where your Redis Cloud database is deployed.
    • azs: The availability zone IDs where your Redis Cloud database is deployed.
    • port: The port number for the new PostgreSQL source database.
    • name: A prefix for all of the created AWS resources.
    • redis_secrets_arn: The role ARN from Secrets in the Redis Cloud console.
    • redis_privatelink_arn: The role ARN from Source connectivity in the Redis Cloud console.

  5. To view the configuration, run:

    terraform plan -var-file=example.tfvars

  6. To create the AWS resources, run:

    terraform apply -var-file=example.tfvars

    This example creates the following resources on your AWS account:

    • An AWS KMS key with the required permissions for RDI
    • A VPC with a public and private subnet and all necessary route tables
    • An EC2 instance running a PostgreSQL database with a security group that allows access from Redis Cloud
    • An AWS Secrets Manager secret for the PostgreSQL database credentials
    • A Network Load Balancer (NLB), a listener, and target group to route traffic to the EC2 instance with AWS PrivateLink
    • An AWS PrivateLink endpoint service for the PostgreSQL database

Creating the AWS resources will take some time. After the resources are created, you'll be able to view them in the AWS management console.

Save the following outputs:

  • database: The name of the PostgreSQL database.
  • port: The port number for the PostgreSQL database.
  • secret_arn: The ARN of the AWS Secrets Manager secret for the PostgreSQL database credentials.
  • vpc_endpoint_service_name: The name of the AWS PrivateLink endpoint service for the PostgreSQL database.

If you lose any outputs, run terraform output to view them again.

Resume pipeline setup

  1. Return to your database's Data Integration tab in the Redis Cloud console.

  2. Open the draft pipeline's actions menu and select Resume pipeline setup.

  3. Open Configure source and select your PostgreSQL source in the Sources list.

  4. Under Source connectivity, enter the Terraform vpc_endpoint_service_name output as the Private Link service name.

  5. Select Connect to Private Link and wait for connectivity to complete.

    AWS Private Link connectivity with the service name and Connect to Private Link control.

  6. Under Secrets, enter the secret_arn output as Credentials Secret ARN.

  7. Select Validate to check access to the secret.

    The Credentials Secret ARN field, transit security options, and Validate control.

  8. Under Source configuration, enter the database and port Terraform outputs in the matching fields.

  9. Select Test source and correct any validation errors, then select Continue.

  10. In Select data, select the schemas, tables, and columns to ingest. Review the selected key for each table.

    Selecting a schema shows its tables for ingestion. Selecting a table shows its columns and the columns selected for ingestion.

  11. Select Continue to open Add transformations. For this example, you can keep the default mapping without adding jobs.

  12. Select Continue to review & deploy.

  13. Review the source and target, then select Deploy pipeline.

    The Deploy pipeline button.

The source first imports its selected data, then captures ongoing changes. Open the pipeline's Dashboard or Metrics tab to follow its progress.

The following example shows the metrics for one selected source in a pipeline with multiple sources.

Metrics for one selected source in a pipeline, including snapshot progress and per-table record counts.

See View and edit data pipeline for source actions, dataset changes, and monitoring.

Delete sample resources

Warning:
Make sure to delete your data pipeline before deleting the sample resources.

To delete the sample resources created by Terraform, run:

terraform destroy -var-file=example.tfvars
RATE THIS PAGE
Back to top ↑