BuildStockQuery is a python library designed to simplify and streamline the process of querying massive, terabyte-scale datasets generated by ResStock™ and ComStock™. ResStock (SWR-19-15) is a U.S. DOE-supported, NREL-built, national residential building energy stock model that enables a new approach to large-scale residential energy analysis across the U.S. by combining large public and private data sources, statistical sampling, detailed sub-hourly building simulations, and high-performance computing.
BuildStockQuery offers an intuitive Object-Oriented Programming (OOP) interface to the ResStock output dataset allowing users to easily perform common queries and receive results in familiar pandas DataFrame format, abstracting away the need for complex SQL query. By initializing a query object with the pertinent Athena database and table names, users can easily query for various kinds of insights, for example, timeseries electricity for an end use for a given state grouped by building types.
Documentation and walkthrough: https://github.com/NREL/buildstock-query/wiki
Usage examples: https://github.com/NREL/buildstock-query/tree/main/example_usage
Standalone CLI utilities to help with AWS Athena table maintenance and search.
Programmatic usage examples: example_usage/aws_athena_tools.ipynb
Scan an Athena database and identify (or drop) stale tables and views whose underlying S3 data no longer exists. If --drop is used and all tables/views in the database are stale, the empty database itself is also dropped.
# Dry-run: report stale tables/views without dropping
aws_athena_cleanup --database my_db --workgroup primary --region us-west-2
# Actually drop stale objects (drops the database too if it becomes empty)
aws_athena_cleanup --database my_db --workgroup primary --drop
# Skip view inspection (tables only)
aws_athena_cleanup --database my_db --workgroup primary --skip-views
# Help: see available configuration setting
aws_athena_cleanup --helpSearch for a table across all Glue/Athena databases in your catalog.
# Exact match across all databases
aws_athena_table_search --table my_table_name
# Substring match (find tables containing "baseline")
aws_athena_table_search --table baseline --substring
# Regex match
aws_athena_table_search --table "baseline_\d+" --regex
# Only search databases whose name contains "resstock"
aws_athena_table_search --table my_table --database-filter resstock
# Different region
aws_athena_table_search --table my_table --region us-east-1
# Help: see available configuration setting
aws_athena_table_search --help