How the Open Water Foundation is Improving Workflow Automations with Open-Source Software
Steve Malers, Chief Executive Officer and Chief Technology Officer, Open Water Foundation
September 2024
To make data-informed decisions, organizations need their data to be clean, organized, and accessible. While time-consuming, processing water data into information products is critical for many organizations to support their operations and decision-making. Automating workflows to process datasets can increase efficiency, but these automations are frequently complex. Information products and decisions may be internally or externally facing, and cloud platforms are increasingly used to store private and public data. Many “software as a service” (SaaS) platforms are now available to host an organization’s content and help automate work; however, there is a financial cost due to the subscription payment model and SaaS platforms often require technical skills and ongoing investment. Combining efficient automated workflows with appropriate SaaS solutions can provide cost-effective solutions.
The Open Water Foundation (OWF) has been on a long journey to develop and implement open-source software tools that convert complex datasets into useful information products that can be used for data-informed decision-making. OWF was born out of a need to help the State of Colorado maintain essential complex software tools used for water resources modeling and decision support. Now, the OpenCDSS initiative is a hub of the open-source tools used by water data entities nationwide.
Example of TSTool interface listing Colorado’s 2,542 gages and their daily time series
TSTool: A Workhorse for Water Data Processing
OWF originally developed software to process data into formats needed for the State of Colorado’s models, which at the time required significant expertise in both software and water resources, as well as ongoing funding for maintenance and enhancements. In the process, OWF realized that some of these tools had value outside the specific modeling environment.
One of these tools, the Time Series Tool (TSTool), was originally developed in 1994 and continues to be a workhorse for automating data processing workflows. OWF actively supports the core TSTool software, which integrates with important water data sources such as the US Geological Survey (USGS) and state agencies. Additionally, OWF has implemented a plugin capability to allow additional data sources and commands to be added to the core product.
Example of TSTool graph view created from Reclamation spreadsheet data read using a URL
Enhancing Workflow Efficiency with TSTool
TSTool provides 300 built-in commands to process time series, tables, spreadsheets, databases, spatial data layers, visualizations, and other data, and includes commands for logic decisions and data manipulation. The software handles the “nuts and bolts” like missing data, errors, and unit compatibility that should be simple but are often neglected in other software. TSTool also includes features for quality control, automate workflow testing, and interactive data visualizations, all useful for water resources management. This free software is available for both Windows and Linux and can run in interactive or batch mode. Workflows use a simple text-based language that can be edited in a text editor or TSTool’s graphical command editor and also integrates with Python and R, further extending its functionality.
Most projects and business processes can be divided into datasets (e.g., hosted in the cloud as files or web services), workflows (e.g., TSTool workflow), and information products (e.g., graph or map, viewed locally or published to the cloud). TSTool handles many of the tasks necessary for creating a complete workflow, helping organizations implement efficient, transparent, cost-effective, and sustainable workflows. The software is used by many organizations, including the State of Colorado’s water agencies, Denver Water, the US Bureau of Reclamation, and others.
By creating and maintaining free, open-source tools like TSTool, OWF continues to support the water data community in making transparent, data-informed decisions about our water.
Example web application that uses time series data uploaded by TSTool
Learn more
Contact Steve Malers at the Open Water Foundation to learn more about how you can use TSTool in your organization.