Introduction

Exported data is placed in a dedicated GCP storage bucket. This document explains how you can access and import data with Data Export from GCP. Although customers may opt for platform such as GCP, AWS and, Azure, this document provides a guide for customers using AWS and GCP. If you need a general guide to bulk data export, please read Bulk Data Export first.

Keys Sharing

For a seamless data transfer from GCP, it is essential to have a service account granted with the necessary permissions and its corresponding key for authentication. This ensures that the workload has access to GCP, minimizing the risk of permission-related issues. However, different keys needed depend on the way you use them. According to your convenience, we can provide the following two types of keys.

Option 1. Service Account Key

A service account key is a JSON file that serves as an authentication credential for a service account and lets a workload authenticate as a service account. The service account key includes information such as the private key, client email, and other metadata. This key is used to authenticate and authorize requests made by your applications or services when interacting with GCP resources and APIs. If you want to know how to set up service account key in your local environment, please see How to set up Application Default Credential with service account key.

Data Transfer Example:

• How to transfer data from GCS to Databricks with service account key

📘

Note

Service account keys are a security risk if not managed correctly, so you are responsible for the security of the private key. For more information, see best practices for managing service account keys.

Option 2. HMAC Key

An HMAC key is a type of credential and can be associated with a service account or a user account in Cloud Storage. You use an HMAC key to create signatures which are then included in requests to Cloud Storage. Most importantly, the HMAC key has the same key format as S3, customers only need to change the endpoint config from AWS S3 to GCS in the SDK settings. If you want to see migration on S3 SDK, please see migration from Amazon S3 to Cloud Storage.

HMAC key contains two parts:

1. Access ID: An alphanumeric string linked to a specific service or user account. When linked to a service account, the string is 61 characters in length, and when linked to a user account, the string is 24 characters in length.
2. Secret: A 40-character Base-64 encoded string that is linked to a specific access ID. A secret is a shared key that only you and Cloud Storage know. You use your secret to create signatures as part of the authentication process.

Both the access ID and secret uniquely identify an HMAC key, but the secret is much more sensitive information, which can not be retrieved anymore once you lose it. For how to store secrets, please see best practices for storing secrets.

Data Transfer Example:

• How to access Google Cloud Storage using AWS S3 endpoint

📘

Note

If you need the above keys for transferring data, please contact your Moloco representative.

Diagram