Using Amazon S3 as a snapshot repository

CrateDB supports using the Amazon S3 (Amazon Simple Storage Service) as a snapshot repository. For this, you need to register the AWS plugin with CrateDB.

Basic configuration

Support for Snapshot and Restore to the Amazon S3 service is enabled by default in CrateDB. If you need to explicitly turn it off, disable the cloud setting in the crate.yml file:

cloud.enabled: false

To be able to use the S3 API, CrateDB must sign the requests by using AWS credentials consisting of an access key and a secret key. Therefore AWS provides IAM roles to avoid any distribution of your AWS credentials to the instances.

Authentication

It is recommended to restrict the permissions of CrateDB on S3 to only the required extent. First, an IAM role is required. This AWS IAM policy guide explains how to create a policy by using the CLI or the AWS Management Console. Further, access of the snapshot to the S3 bucket needs to be restricted. An example policy file granting anybody access to a bucket called snaps.example.com is attached below:

{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Principal": "*",
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Principal": "*",
      "Resource": [
        "arn:aws:s3:::snaps.example.com/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

Access permissions can be further restricted to a specific AWS Principal by changing the Statement.Principal setting. Please refer to AWS principals for more information.

For further AWS policy examples and detailed information, please refer to AWS policy examples and the links therein.

It has to be noted, that the bucket needs to exist before registering a repository for snapshots within CrateDB. CrateDB can also be allowed to create the bucket. However this requires the following permissions to be contained within the policy:

{
   "Action": [
      "s3:CreateBucket"
   ],
   "Effect": "Allow",
   "Resource": [
      "arn:aws:s3:::snaps.example.com"
   ]
}