Opportunities and Challenges for Data, Models, Computation, and Workflows
This article highlights use cases of ocean observation to explore how cloud computing can be improved to handle increased data flows. As the amount of data ingested increases, the cloud could replace traditional approaches to data warehousing. High-performance mass storage of observational data, coupled with on-demand computing to run model simulations near the data, tools to manage workflows, and a framework to share and collaborate, enables a more flexible and adaptable observation and prediction computing architecture. Apply this structure in your industry regarding how to get data, store data, organize it, and conduct analysis and visualization in the cloud. What are some potential problems for large datasets? Think about how you would overcome those challenges. How would "sandboxes" provide some security when testing a system?
Defining Cloud Computing and Patterns for Its Use
The most widely used definition of cloud computing is in Mell and Grance:
Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.
Essential Characteristics:
• On-demand self-service: A consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed.
• Broad network access: Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms.
• Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model.
• Rapid elasticity: Capabilities can be elastically provisioned and released, in some cases automatically, to scale rapidly outward and inward commensurate with demand.
• Measured service: Cloud systems automatically control and optimize resource use by leveraging a metering capability.
Service Models:
• Software as a Service (SaaS): The capability provided to the consumer is to use the provider's applications running on a cloud infrastructure.
• Platform as a Service (PaaS): The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages, libraries, services, and tools supported by the provider.
• Infrastructure as a Service (IaaS): The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software.
For this paper, we define the cloud as shared, off-premises user configurable resources for data storage/discovery and computing.
Butler and Merati provide another view of the cloud. Following the framework of A Pattern Language and applications of patterns to object-oriented programming in Gamma et al., they define six patterns of cloud use:
Cloud-Based Scientific Data – Getting Data From the Cloud
Intent: Explores integrating the use of cloud-based data and how scientists can access large volumes of diverse, current and authoritative data, addresses the problem of locating and using large amounts of scientific data.
The Section "Architectures for Real-Time Data Management and Services for Observations" describes streaming data and an architecture for making it easy to gather data. Also see Johanson et al.
Cloud-Based Management of Scientific Data – Storing Data in the Cloud
Intent: Explores storing and managing data in the cloud. Addresses the problem of ever increasing data quantities with decreasing budgets for data management. Explores the ways scientific projects can meet data access and dissemination requirements such as the U.S. Public Access to Research Results (PARR) mandate.
The section on the NOAA Big Data Project and open data and archiving are examples of this pattern. Also see Meisinger et al.
Computing Infrastructure for Scientific Research
Intent: Explores the ways in which cloud computing, in the form of PaaS or IaaS could be used as part of a research program and for teaching. It addresses the need for larger computational capabilities, especially under constrained budgets.
The modeling efforts described in later sections are examples of this pattern.
Analysis in the Cloud
Intent: Explores conducting analyses in the cloud. Addresses the problem of wanting to perform analyses on ever larger datasets and on datasets from multiple sources. Explores the secondary question of ways scientific projects can standardize analysis tools among geographically distributed researchers.
The section entitled "Architectures for Real-Time Data Management and Services for Observations" is an example of this pattern, as are Henderson and Gorelick et al.
Visualization
Intent: Explores creating visualizations using cloud-based tools and making the visualizations available via the cloud. Addresses the need to visualize larger amounts of data and the opportunities provided by improved graphics processors and display devices such as VR headsets.
The section entitled "Workflow on the cloud" shows examples of this pattern as does Allam et al. Workflow tools can visualize more than just the data, they can reveal unexpected dependencies, bottlenecks, and participants' roles.
Results Dissemination in Real Time/Storytelling/Outreach
Intent: Explores ways in which cloud-based platforms and tools can be used to reach new audiences. Addresses the need to make research results rapidly available and relevant to a wide variety of audiences – scientific and non-scientific.
The US Integrated Ocean Observing System (IOOS) Regional Associations' work described below are examples of this pattern.