Feb 07 20244 min.

Hybrid storage solutions to meet business requirements on AWS

Overview of Use Cases and Key Features of Hybrid Storage Solutions on AWS

Full-Stack Cloud Engineer

TABLE OF CONTENTS

AWS Storage Gateway
AWS DataSync
AWS Snow Family
References

Throughout this article, I mentioned and referred to various storage types available on AWS. If you need a quick refresher, please refer to the previous article titled "Storage Types with Associated Characteristics in AWS."

Hybrid storage is one of the four storage types available on AWS, the remaining three being object, block, and file storage. Actually, hybrid storage differs from the rest in that it can integrate any of these three types of storage with on-premises systems, hence the name hybrid.

The dedicated hybrid solution on AWS is AWS Storage Gateway, which comes in different flavors: File Gateway, Volume Gateway, and Tape Gateway.

Apart from AWS Storage Gateway, various storage services and tools offer to some extent hybrid features, like AWS DataSync, AWS Snow Family, Amazon Elastic File System (EFS).

AWS Storage Gateway

As mentid in introduction a truelly hybrid storage that can integrate on-premise and AWS cloud storage solutions covering various use cases hence three different types of this service which are cover below. It's worth to notice that it can be used as a VM or hardware appliance.

File Gateway

Related to file-based storage, the concept of File Gateway is to extend your local (on-premise) storage with cloud storage, which can be either Amazon S3 or Amazon FSx for Windows File Server, depending on the protocol your environment utilizes.

File Gateway helps seamlessly integrate with the aforementioned AWS cloud storage services, providing all the benefits of their services such as scalability, durability, availability, and cost-effectiveness.

Typically, File Gateway is used for data archiving, backup, and making on-premises data available for cloud-based processing and analytics.

A notable feature is local caching for frequently accessed files, reducing costs and enhancing performance.

It utilizes NFS (for Linux) and SMB (for Windows) protocols which are mainly designed for sharing files over network.

Volume Gateway

Volume Gateway differs from File Gateway in that it uses the iSCSI protocol, which is primarily utilized in Storage Area Networks (SANs) with block storage instead of file storage.

Similar to File Gateway, Volume Gateway extends local storage into the AWS cloud, enabling the storage and retrieval of data over the network. It can operate in two different modes depending on requirements.

Stored Volumes

In this mode, you store data locally while asynchronously sending data to AWS in the form of EBS snapshots. All data is accessed locally, and the data sent to AWS serves as a backup or for disaster recovery purposes.

Cached Volumes

This mode is the opposite of Stored Volumes. Most data is accessed over the network from Amazon S3, while the most frequently accessed data is cached locally to enhance performance and reduce costs.

Tape Gateway

Tape Gateway aims at environments that utilize physical tape-based backup systems and want to continue using the same tape-based software but with the benefits of cloud storage. To mimic physical tapes, it utilizes a Virtual Tape Library (VTL) and stores data in Amazon S3 Glacier Flexible Retrieval or Amazon S3 Glacier Deep Archive.

Similar to traditional tape-based solutions, Tape Gateway primarily serves as an archiving and backup solution. However, you can't access data directly after it's stored on S3 Glacier due to the characteristics of S3 Glacier. Accessing or retrieving the stored data requires using the AWS Storage Gateway console or the AWS Storage Gateway API.

AWS DataSync

AWS DataSync is not a hybrid storage service, but it plays a crucial role in hybrid storage architectures. The service facilitates data transfer between on-premises storage systems and AWS services, as well as between AWS services themselves.

It's commonly used for data migration because it can greatly accelerate data transfers through parallel transfers, supports integration with a plethora of storage systems, can analyze local storage to better plan migration, which can be fully automated and scheduled.

Utilizing this service is a good idea when first migrating data to AWS.

AWS Snow Family

The AWS Snow Family can be considered as a hybrid storage solution due to its computing capabilities, such as those offered by Snowball Edge. However, it is primarily designed for the physical transfer of data into and out of AWS. It is available in three main configurations (with variations within the Snowball category):

Snowcone

Supports up to 14 TB of data and is suitable for edge computing and data transfer in environments with limited connectivity.

Snowball

Available in different versions to serve various needs, including storage-optimized and compute-optimized options, with capacities of up to 80 TB for storage-optimized and 42 TB for compute-optimized Snowball Edge devices.

Snowmobile

A truck-mounted solution capable of migrating up to 100 PB of data, ideal for extremely large-scale data center migrations to AWS.

Common use cases include transferring large amounts of data securely, especially where the network is insecure or weak. The entire Snow Family is rugged, designed to securely transport data under various environmental conditions.