CREATE REPOSITORY
¶
You can use the CREATE REPOSITORY
statement to
register a new repository that you can use to create, manage, and restore
snapshots.
See also
Table of contents
Synopsis¶
CREATE REPOSITORY repository_name TYPE type
[ WITH (parameter_name [= value], [, ...]) ]
Description¶
The CREATE REPOSITORY
statement creates a repository with a
repository name and repository
type. You can configure the different types of repositories WITH
additional parameters.
Note
If the back-end data storage (specific to the repository type) already contains CrateDB snapshots, they will become available to the cluster.
See also
Parameters¶
- repository_name
The name of the repository to register.
- type
The repository type.
Caution
You cannot change any repository parameters after creating the repository (including parameters set by the WITH clause).
Suppose you want to use new parameters for an existing repository. In that
case, you must first drop the repository using the DROP REPOSITORY statement and then recreate it with a new CREATE
REPOSITORY
statement.
When you drop a repository, CrateDB deletes the corresponding record from sys.repositories but does not delete any snapshots from the corresponding backend data storage. If you create a new repository using the same backend data storage, any existing snapshots will become available again.
Clauses¶
WITH
¶
You can use the WITH
clause to specify one or more repository parameter
values:
[ WITH (parameter_name [= value], [, ...]) ]
Parameters¶
The following parameters apply to all repository types:
- max_restore_bytes_per_sec
The maximum rate (bytes per second) at which a single CrateDB node will read snapshot data from this repository.
Default:
40mb
- max_snapshot_bytes_per_sec
The maximum rate (bytes per second) at which a single CrateDB node will write snapshot data to this repository.
Default:
40mb
All other parameters (see the next section) are specific to the repository type.
Types¶
CrateDB includes built-in support for the following types:
CrateDB can support additional types via plugins.
fs
¶
An fs
repository stores snapshots on the local file system. If a cluster
has multiple nodes, you must use a shared data storage volume mounted locally
on all master nodes and data nodes.
Note
To create fs
repositories, you must configure the list of allowed file
system paths using the path.repo setting.
Parameters¶
- location
- Type:
text
RequiredAn absolute or relative path to the directory where CrateDB will store snapshots. If the path is relative, CrateDB will append it to the first entry in the path.repo setting.
Windows UNC paths are allowed if the server name and shares are specified and backslashes are escaped.
The path must be allowed by the path.repo setting.
- compress
- Type:
boolean
Default:true
Whether CrateDB should compress the metadata part of the snapshot or not.
CrateDB does not compress the actual table data.
- chunk_size
- Type:
bigint
ortext
Default:null
Defines the maximum size of any single file that comprises the snapshot. If set to
null
, CrateDB will not split big files into smaller chunks. You can specify the chunk size with units (e.g.,1g
,5m
, or9k
). If no unit is specified, the unit defaults to bytes.
s3
¶
An s3
repository stores snapshot on the Amazon Simple Storage Service
(Amazon S3).
Note
If you are using Amazon S3 in conjunction with IAM roles, the
access_key
and secret_key
parameters must be left undefined.
Additionally, make sure to attach the IAM to each EC2 instance that will run a CrateDB master node or data node. The attached IAM role will provide the necessary credentials when required.
Parameters¶
- access_key
- Type:
text
Required:false
Access key used for authentication against Amazon Web Services (AWS).
Note
CrateDB masks this parameter. You cannot query the parameter value from the sys.repositories table.
- secret_key
- Type:
text
Required:false
The secret key used for authentication against AWS.
Note
CrateDB masks this parameter. You cannot query the parameter value from the sys.repositories table.
- endpoint
- Type:
text
Default: The default AWS API endpointThe AWS API endpoint to use.
Tip
You can specify a regional endpoint to force the use of a specific AWS region.
- protocol
- Type:
text
Values:http
,https
Default:https
Protocol to use.
- bucket
- Type:
text
Name of the Amazon S3 bucket used for storing snapshots.
If the bucket does not yet exist, CrateDB will attempt to create a new bucket on Amazon S3.
- base_path
- Type:
text
Default:root directory
The bucket path to use for snapshots.
The path is relative, so the
base_path
value must not start with a/
character.
- compress
- Type:
boolean
Default:true
Whether CrateDB should compress the metadata part of the snapshot or not.
CrateDB does not compress the actual table data.
- chunk_size
- Type:
bigint
ortext
Default:null
Defines the maximum size of any single file that comprises the snapshot. If set to
null
, CrateDB will not split big files into smaller chunks. You can specify the chunk size with units (e.g.,1g
,5m
, or9k
). If no unit is specified, the unit defaults to bytes.
- readonly
- Type:
boolean
Default:false
If
true
, the repository is read-only.
- server_side_encryption
- Type:
boolean
Default:false
If
true
, files are server-side encrypted by AWS using theAES256
algorithm.
- buffer_size
- Type:
text
Default:5mb
Minimum:5mb
If a chunk is smaller than
buffer_size
, CrateDB will upload the chunk with a single request.If a chunk exceeds
buffer_size
, CrateDB will split the chunk into multiple parts ofbuffer_size
length and upload them separately.
- max_retries
- Type:
integer
Default:3
The number of retries in case of errors.
- use_throttle_retries
- Type:
boolean
Default:true
Whether CrateDB should throttle retries (i.e., should back off).
- canned_acl
- Type:
text
Values:private
,public-read
,public-read-write
,authenticated-read
,log-delivery-write
,bucket-owner-read
, orbucket-owner-full-control
Default:private
When CrateDB creates new buckets and objects, the specified Canned ACL is added.
- storage_class
- Type:
text
Values:standard
,reduced_redundancy
orstandard_ia
Default:standard
The S3 storage class used for objects stored in the repository. This only affects the S3 storage class used for newly created objects in the repository.
- use_path_style_access
- Type:
boolean
Default:false
Whether CrateDB should use path style access. Useful for some S3-compatible providers.
azure
¶
An azure
repository stores snapshots on the Azure Blob storage service.
Parameters¶
- account
- Type:
text
The Azure Storage account name.
Note
CrateDB masks this parameter. You cannot query the parameter value from the sys.repositories table.
- key
- Type:
text
The Azure Storage account secret key.
Note
CrateDB masks this parameter. You cannot query the parameter value from the sys.repositories table.
- sas_token
- Type:
text
The Shared Access Signatures (SAS) token used for authentication for the Azure Storage account. This can be used as an alternative to the Azure Storage account secret key.
The SAS token must have read, write, list, and delete permissions for the repository base path and all its contents. These permissions need to be granted for the blob service and apply to resource types service, container, and object.
Note
CrateDB masks this parameter. You cannot query the parameter value from the sys.repositories table.
- endpoint
- Type:
text
The Azure Storage account endpoint.
Tip
You can use an sql-create-repo-azure-endpoint to access Azure Storage instances served on private endpoints.
Note
endpoint
cannot be used in combination with sql-create-repo-azure-endpoint_suffix.
- secondary_endpoint
- Type:
text
The Azure Storage account secondary endpoint.
Note
secondary_endpoint
cannot be used if sql-create-repo-azure-endpoint is not specified.
- endpoint_suffix
- Type:
text
Default:core.windows.net
The Azure Storage account endpoint suffix.
Tip
You can use an endpoint suffix to force the use of a specific Azure service region.
- container
- Type:
text
Default:crate-snapshots
The blob container name.
Note
You must create the container before creating the repository.
- base_path
- Type:
text
Default:root directory
The container path to use for snapshots.
- compress
- Type:
boolean
Default:true
Whether CrateDB should compress the metadata part of the snapshot or not.
CrateDB does not compress the actual table data.
- chunk_size
- Type:
bigint
ortext
Default:256mb
Maximum:256mb
Minimum:1b
Defines the maximum size of any single file that comprises the snapshot. If set to
null
, CrateDB will not split big files into smaller chunks. You can specify the chunk size with units (e.g.,1g
,5m
, or9k
). If no unit is specified, the unit defaults to bytes.
- readonly
- Type:
boolean
Default:false
If
true
, the repository is read-only.
- location_mode
- Type:
text
Values:primary_only
,secondary_only
,primary_then_secondary
,secondary_then_primary
Default:primary_only
The location mode for storing blob data.
Note
If you set
location_mode
tosecondary_only
,readonly
will be forced totrue
.
- max_retries
- Type:
integer
Default:3
The number of retries (in the case of failures) before considering the snapshot to be failed.
- timeout
- Type:
text
Default:30s
The client side timeout for any single request to Azure.
- proxy_type
- Type:
text
Values:http
,socks
, ordirect
Default:direct
The type of proxy to use when connecting to Azure.
- proxy_host
- Type:
text
The hostname of the proxy.
- proxy_port
- Type:
integer
Default:0
The port number of the proxy.
gcs
¶
A gcs
repository stores snapshots on the Google Cloud Storage service.
Parameters¶
- bucket
- Type:
text
RequiredName of the Google Cloud Storage bucket used for storing snapshots. The bucket must already exist before the repository is created.
- private_key_id
- Type:
text
RequiredThe Private key id for the Google Service account from the json Google Service account credentials.
Note
This parameter will be masked (shown as
[xxxxx]
) when querying sys.repositories table.
- private_key
- Type:
text
RequiredThe private key in PKCS 8 format for the Google Service account from the json Google Service account credentials.
Note
This parameter will be masked (shown as
[xxxxx]
) when querying sys.repositories table.
- client_id
- Type:
text
RequiredThe client id for the Google Service account from the json Google Service account credentials.
Note
This parameter will be masked (shown as
[xxxxx]
) when querying sys.repositories table.
- client_email
- Type:
text
RequiredThe client email for the Google Service account from the json Google Service account credentials.
Note
This parameter will be masked (shown as
[xxxxx]
) when querying sys.repositories table.
- base_path
- Type:
text
Default:root directory
The container path to use for snapshots.
- compress
- Type:
boolean
Default:true
Whether CrateDB should compress the metadata part of the snapshot or not.
- chunk_size
- Type:
bigint
ortext
Default:null
Defines the maximum size of any single file that comprises the snapshot. If set to
null
, the default value 5 Terabyte is used. You can specify the chunk size with units (e.g.,1g
,5m
, or9k
). If no unit is specified, the unit defaults to bytes.
- connect_timeout
- Type:
text
Default:0
Defines the timeout to establish a connection to the Google Cloud Storage service. The value should specify the unit. For example, a value of 5s specifies a 5 second timeout. The value of -1 corresponds to an infinite timeout. The default value
0
indicates to use the default value of20s
from the Google Cloud Storage library.
- read_timeout
- Type:
`text
Default:0
Defines the timeout to read data from an established connection. The value should specify the unit. For example, a value of 5s specifies a 5 second timeout. The value of -1 corresponds to an infinite timeout. The default value
0
indicates to use the default value of20s
from the Google Cloud Storage library.
- endpoint
- Type:
text
Required:false
Endpoint root url to connect to an alternative storage provider.
- token_uri
- Type:
text
Required:false
Endpoint oauth token URI to connect to an alternative oauth provider.
url
¶
A url
repository provides read-only access to an fs repository via one of the supported network access
protocols.
You can use a url
repository to restore snapshots.
Parameters¶
- url
- Type:
text
The root URL of the fs repository.
Note
The URL must match one of the URLs configured by the repositories.url.allowed_urls setting.