Migrate data to Object Storage using rclone

By the end of this tutorial, rclone is configured with a source and a destination storage endpoint and data is migrated between them. This tutorial uses STACKIT Object Storage as the migration destination.

Prerequisites

• rclone installed on your system.
• Object Storage access credentials for the destination endpoint. For instructions, read Create Object Storage credentials.
• Access credentials for the source endpoint.

Configure source and destination

rclone uses a configuration file at ~/.config/rclone/rclone.conf to store remote definitions. Open this file in a text editor and add entries for both the source and destination, replacing the placeholder values with your actual credentials:

[source]
type = s3
provider = Other
access_key_id = YOUR_SOURCE_ACCESS_KEY
secret_access_key = YOUR_SOURCE_SECRET_KEY
endpoint = https://platform.cloud.schwarz:8080
acl = private

[destination]
type = s3
provider = Other
access_key_id = YOUR_DESTINATION_ACCESS_KEY
secret_access_key = YOUR_DESTINATION_SECRET_KEY
endpoint = https://object.storage.eu01.onstackit.cloud
acl = private

Verify that both remotes are registered correctly:

rclone config

The output lists all configured remotes:

Current remotes:

Name                 Type
====                 ====
destination          s3
source               s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q>

Encrypt your rclone configuration file to protect the access and secret keys stored within it. Select Set configuration password from the rclone config menu, choose Add Password, and follow the prompts to set a password.

Migrate the data

With both remotes configured, copy the data from the source bucket to the destination bucket. Replace BUCKET_NAME with your actual bucket name:

rclone copy source:/BUCKET_NAME destination:/BUCKET_NAME \
  --s3-upload-cutoff 0 \
  --progress \
  --log-file /tmp/rclone.log

rclone reports progress during the transfer and writes detailed output to /tmp/rclone.log. When the copy finishes, the summary shows the number of bytes and files transferred:

Transferred:       300 / 300 Bytes, 100%, 218 Bytes/s, ETA 0s
Errors:                  0
Checks:                  0 / 0, -
Transferred:            50 / 50, 100%
Elapsed time:          1.3s

After the transfer completes, review the log file for any errors:

cat /tmp/rclone.log

Next steps

For a full reference of all rclone flags and options, see the official rclone documentation.