Read on Twitter
Let's look at options for data sharing in the cloud and how things might be shifting more and more in the future.
This is relevant for many organizations facing the challenges of data movement cost, speed of access, or the risks of trying out new and promising approaches.
1/n
This is relevant for many organizations facing the challenges of data movement cost, speed of access, or the risks of trying out new and promising approaches.
1/n
Data across clouds comes into play as well.
With improving interoperability between technologies, many companies want to choose the best services and products to solve their scenarios, even if they are services on different clouds. E.g. Cloud X is good for A, Cloud Y for B.
2/n
With improving interoperability between technologies, many companies want to choose the best services and products to solve their scenarios, even if they are services on different clouds. E.g. Cloud X is good for A, Cloud Y for B.
2/n
This leads to data distributed over different services, regions or clouds (take even logging information on diff clouds). Organizations still want to be able to query data efficiently & with minimal costs.
There are challenges and trade-offs. Let's look at common options.
3/n
There are challenges and trade-offs. Let's look at common options.
3/n
First, one of the common options.
Several services are running in the same cloud. Compute and storage isn't independent. To work with the data from another service we'd have to copy or move it. Nothing wrong for simple architectures, if this meets your criteria.
4/n
Several services are running in the same cloud. Compute and storage isn't independent. To work with the data from another service we'd have to copy or move it. Nothing wrong for simple architectures, if this meets your criteria.
4/n
Second - quickly gaining adoption.
Several services in the same cloud. Compute & storage are completely separate. Better scalability, cost-efficiency, direct access to data. Ex: Azure Synapse Link for Azure Cosmos DB, BigQuery external data sources, Snowflake data sharing.
5/n
Several services in the same cloud. Compute & storage are completely separate. Better scalability, cost-efficiency, direct access to data. Ex: Azure Synapse Link for Azure Cosmos DB, BigQuery external data sources, Snowflake data sharing.
5/n
Third - common in third-party managed systems, or systems you manage that span multiple clouds and/or regions.
Consumers can access data from both clouds, requires replication (cost, time) even if it's happening under-the-hood. E.g. cross-region data sharing in Snowflake.
6/n
Consumers can access data from both clouds, requires replication (cost, time) even if it's happening under-the-hood. E.g. cross-region data sharing in Snowflake.
6/n
Fourth:
multiple clouds
storage across multiple different-cloud services separation of compute and storage
direct data access - no need to copy or move data
Enabled by technologies like Azure Arc or Google Anthos. E.g. BigQuery Omni, Azure Arc-enabled data services
7/n
multiple cloudsstorage across multiple different-cloud services separation of compute and storagedirect data access - no need to copy or move dataEnabled by technologies like Azure Arc or Google Anthos. E.g. BigQuery Omni, Azure Arc-enabled data services
7/n
There are two themes that are true for enabling better data sharing options for an organization:
separation of storage and compute, link to other data sources for direct access.
multi-cloud or hybrid platforms that bring compute to data located anywhere (Anthos, Arc).
8/n
separation of storage and compute, link to other data sources for direct access.multi-cloud or hybrid platforms that bring compute to data located anywhere (Anthos, Arc).8/n
For cloud providers this means, companies might not be choosing some cloud services because they have already committed to a certain cloud and are storing data there.
These multi-cloud technologies still need to develop and mature, but they are really promising and useful.
9/n
These multi-cloud technologies still need to develop and mature, but they are really promising and useful.
9/n
If companies can store data anywhere with easy access & sharing, they will also adjust decision-making.
Developer experience, programmability, interoperability, integrations, open standards - I think these factors will become more prevalent when choosing cloud services.
10/n
Developer experience, programmability, interoperability, integrations, open standards - I think these factors will become more prevalent when choosing cloud services.
10/n
Convenient services supporting data sharing with tunable access also helps with eliminating silo-teams and monster-data-lakes, enabling better data sharing options for domain-organized data-product teams, per @zhamakd's data mesh concept.
/end
/end
