How to Transfer Data
Introduction
Exported data is placed in a dedicated Google Cloud Storage (GCS) bucket. This document explains the available methods for securely accessing and transferring your MCM data export to your own cloud environment.
MCM supports several access methods depending on your cloud provider and security requirements. Choose the method that best fits your organization's infrastructure and compliance needs.
Access Methods Overview
| Method | Cloud Provider | Credential Type | Key Management | Best For |
|---|---|---|---|---|
| Dynamic Key Rotation (API) | AWS, Azure, or other non-GCP | Short-lived (Max 48 hours) HMAC | Automated via API | Customers using S3-compatible libraries who want automated credential rotation |
| Workload Identity Federation | AWS, Azure, GCP | Keyless (federated identity) | No keys required | Enterprise customers prioritizing security and compliance |
| GCP Storage Transfer Service | GCP | Google-managed service agent | Managed by Google | Customers already on GCP |
NOTEWhile GCP supports Workload Identity Federation, GCP Storage Transfer Service is the recommended method.
IMPORTANT!To enhance data security and compliance, MCM is transitioning customers using non-GCP cloud providers to one of the two new access methods:
1. Workload Identity Federation OR 2. Dynamic Key RotationThese methods replace the legacy approach of manually sharing long-lived credentials and provide significantly improved security through automated rotation or keyless authentication.
If you are currently using a static HMAC key or service account key, please contact your MOLOCO representative to begin migrating to one of the new methods. Data export access using the legacy manual method will no longer be supported after October 1, 2026.
1. Workload Identity Federation (Recommended)
Workload Identity Federation (WIF) is the most secure access method available. It enables keyless authentication, allowing your AWS or Azure workloads to access the GCS export bucket directly without ever handling a secret key. This is achieved by establishing a trust relationship between your cloud identity provider and Google Cloud.
Prerequisites
- An AWS IAM Role (or Azure equivalent) dedicated to accessing the MCM export bucket.
- The ability to configure your IAM Role's Trust Relationship.
- The Google Cloud SDK (google-cloud-storage v1.27.0 or later for Python).
How it works?
- MOLOCO configures a Workload Identity Pool in GCP and adds a provider linked to your cloud identity (e.g., AWS account).
- MOLOCO provides you with a Credential Configuration file (gcp-creds.json) that contains the metadata needed to bridge your cloud environment with GCP.
- Your application uses the Google Cloud SDK, which automatically performs the identity token exchange in the background — no HMAC keys or service account JSON files are involved.
What benefits do you get?
| Benefit | Description |
|---|---|
| Zero secrets | No keys are generated, stored, or rotated. Eliminates the risk of credential leakage entirely. |
| Industry standard | Follows the OpenID Connect (OIDC) federation model, a widely adopted standard for cross-cloud authentication. |
| Reduced operational overhead | No credential rotation or manual key management required. |
Setup Steps
STEP 1: Identify your Workload Identity
Create (or identify an existing) IAM Role in your cloud environment that will be used to read from the export bucket. For example, in AWS, this could be a role assumed by Lambda, ECS, EC2, or another compute service.
STEP 2: Provide Your Identifier to Moloco
Share the Full Role ARN (for AWS) with your Moloco representative.
arn:aws:iam::123456789012:role/my-data-export-roleFor Azure, provide the equivalent identity information as requested by your Moloco representative.
STEP 3: Prepare Your IAM Role (AWS Example)
Ensure your role's Trust Relationship policy allows the service that will run your data pipeline to assume the role. For example, for an AWS Lambda function:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Principal": {
"Service": "lambda.amazonaws.com"
}
}
]
}
STEP 4: Integrate the Credential Configuration
Moloco will provide a Credential Configuration file (e.g., gcp-creds.json). Place this file in your application's runtime environment and set the following environment variable:
export GOOGLE_APPLICATION_CREDENTIALS="/path/to/gcp-creds.json"
NOTEUse the official Google Cloud SDK (not a generic S3 library) for Workload Identity Federation. The SDK handles the identity token exchange automatically.
STEP 5: Access the Bucket
Using the Google Cloud Storage client library, your application can access the export bucket natively. The SDK automatically reads the credential configuration and performs the cross-cloud token exchange in the background.
Python Example (e.g., AWS Lambda):
import os
from google.cloud import storage
def lambda_handler(event, context):
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "gcp-creds.json"
try:
client = storage.Client()
bucket_name = "mcm-export-your-platform-name"
bucket = client.get_bucket(bucket_name)
blobs = list(client.list_blobs(bucket_name, max_results=5))
print(f"Successfully connected to bucket: {bucket_name}")
print(f"First {len(blobs)} objects:")
for blob in blobs:
print(f" - {blob.name}")
return {
"statusCode": 200,
"body": f"Connected! Found {len(blobs)} objects."
}
except Exception as e:
print(f"ERROR: {str(e)}")
return {"statusCode": 500, "body": str(e)}
NOTEMOLOCO provides the GCP-side configuration (Workload Identity Pool, provider, and service account impersonation). You are responsible for managing your own cloud environment, including the IAM Role and its permissions.
2. Dynamic Key Rotation (API)
Dynamic Key Rotation provides automated, short-lived HMAC credentials through the MCM Management API. Your application calls the RotateDataExportKey API endpoint to obtain fresh credentials before each data access session. Each call invalidates all previously issued keys and returns a new HMAC key pair.
Prerequisites
- Access to the MCM Management API. You must have a valid API key with ADMIN permissions for your platform. If you have not yet onboarded to the Management API, please contact your MOLOCO representative.
How it works?
- Your application calls the RotateDataExportKey endpoint on the MCM Management API.
- The API invalidates all previously issued HMAC keys for your platform's service account.
- A new HMAC key pair (key_id and secret_key) is returned.
- Your application uses the new credentials to access the GCS export bucket via any S3-compatible library or tool.
What benefits do you get?
| Benefit | Description |
|---|---|
| DataEnhanced security | Credentials are effectively short-lived, minimizing the exposure window. |
| Automated rotation | No manual credential management or handoff required. |
| S3 compatibility | Works with existing S3-compatible tooling and libraries; only the credential retrieval step is added. |
IMPORTANT NOTES
- You must call RotateDataExportKey to refresh credentials before every access to the export bucket.
- Each API call must have at least a 1-minute interval from the previous call.
- Newly issued credentials may take up to 60 seconds to propagate and become active. Plan for this delay in your integration.
- The secret_key is returned only once at creation time and cannot be retrieved again. Ensure your application stores it securely for the duration of the data transfer session
- Each call deletes all previously issued keys. Only the most recently issued key pair is valid at any time.
API Reference
Endpoint:
POST /rmp/mgmt/v1/platforms/{platform_id}/data-export/key
Headers:
| Header | Value |
|---|---|
| x-api-key | Your MCM Management API key |
| Content-Type | application/json |
Response:
| Field | Type | Description |
|---|---|---|
| key_id | string | The unique identifier for the newly created HMAC key. |
| secret_key | string | The raw HMAC secret key. Only returned once upon creation. |
NOTEFor full API details, see the MCM Management API Reference.
Integration Example (Python)
The following example demonstrates how to retrieve fresh credentials and use them to access your export bucket.
STEP 1: Retrieve new HMAC credentials
import requests
PLATFORM_ID = "YOUR_PLATFORM_ID"
API_KEY = "YOUR_API_KEY"
BASE_URL = "https://YOUR_PLATFORM-mgmt.rmp-api.moloco.com"
url = f"{BASE_URL}/rmp/mgmt/v1/platforms/{PLATFORM_ID}/data-export/key"
headers = {
"x-api-key": API_KEY,
"Content-Type": "application/json"
}
try:
response = requests.post(url, headers=headers)
response.raise_for_status()
data = response.json()
key_id = data.get("key_id")
secret_key = data.get("secret_key")
print(f"Key ID: {key_id}")
print("Secret Key: (stored securely)")
except requests.exceptions.HTTPError as err:
print(f"HTTP Error: {err}")
except Exception as e:
print(f"Unexpected error: {e}")
STEP 2: Access the GCS bucket with the retrieved credentials
Use the key_id as the Access Key and secret_key as the Secret Key with any S3-compatible library or tool. Since HMAC keys are compatible with the S3 API, you only need to change the endpoint from AWS S3 to GCS (storage.googleapis.com).
For example, using AWS DataSync or S3-compatible SDKs, configure:
- Endpoint: https://storage.googleapis.com/
- Access Key: The key_id value
- Secret Key: The secret_key value
3. GCP Storage Transfer Service
Customers using GCP can transfer data natively using Google Cloud Storage Transfer Service. This method uses a Google-managed service agent — no HMAC keys or service account files are needed.
Prerequisites
- Provide your Storage Transfer Service agent email to your MOLOCO representative so that read access can be granted to your export bucket. The service agent email follows this format:
project-PROJECT_NUMBER@storage-transfer-service.iam.gserviceaccount.com
- A destination GCS bucket in your project.
HMAC Key — Static (Legacy)
Legacy MethodMoloco no longer offers these legacy methods. Existing platforms which use this legacy method migrate to Dynamic Key Rotation or Workload Identity Federation . Please contact your MOLOCO representative to discuss migration options.
Choosing the Right Method
Use the following decision guide to select the best access method for your organization:
- Already on GCP? → Use GCP Storage Transfer Service. No credentials to manage.
- On AWS/Azure and prioritize security? → Use Workload Identity Federation.
- Zero secrets, industry-standard federated identity. On AWS/Azure and prefer S3-compatible tooling?→ Use Dynamic Key Rotation. Minimal integration change with automated credential rotation.
- Currently using static HMAC keys? → Contact your MOLOCO representative to begin migrating to one of the new methods above.
Next steps
- For questions about data export access methods or migration, please contact your MOLOCO representative or see the MCM FAQ.
Updated 17 days ago
