In today’s digital era, businesses rely on cloud computing solutions like BigQuery to make data-driven decisions and gain a competitive edge. However, with the announcement of the upcoming changes to BigQuery pricing, organizations need to be aware of the changes and the potential impact on their cloud spending. Failure to prepare in time could result in budget overruns and financial instability.
This article aims to provide insight into the upcoming BigQuery pricing changes and ways to prepare for them. We will delve into the new BigQuery editions, the benefits they offer, and why understanding the changes is crucial to avoiding potential over-expenditure on cloud services. Additionally, we will discuss two new features, BigQuery Autoscaler and Compressed Storage, that can help organizations optimize their BigQuery usage and reduce costs.
Reasons for GCP BigQuery Pricing Changes
Google implemented these pricing changes to improve the pricing model of BigQuery, making it simpler and more predictable. With the latest information from product documentation and the latest cost information, Economize has developed a BigQuery pricing calculator to help you estimate your BigQuery charges.
The previous pricing model had its challenges, such as the lack of flexibility, which made it difficult for organizations to match their needs with the right pricing plan. Google’s goal with the new pricing model is to provide a more flexible approach that can accommodate the needs of all businesses, regardless of their size or scale.
- Google recently announced changes to BigQuery pricing that will take effect on July 5, 2023.
- The upcoming changes include the removal of flat-rate annual, flat-rate monthly, and flex slot commitments, and a 25% price increase for the on-demand analysis model across all regions.
To provide more value to customers, Google is also introducing two new features with the BigQuery editions.
- BigQuery Autoscaler: adds fine-grained compute resources in real-time to match the needs of your workload demands, ensuring that you only pay for the compute capacity you use.
- Compressed Storage Pricing: allows you to only pay for data storage after it’s been highly compressed. This feature can help reduce your storage costs while increasing your data footprint at the same time.
In the following sections, we will explore the new pricing model and its benefits, as well as discuss ways to prepare for the upcoming changes.
What are BigQuery Pricing Editions?
The new pricing editions are designed to address the challenges that customers faced with the previous pricing tiers. With the new pricing editions, customers can now choose the features and capabilities that meet their specific needs, making it easier to optimize their cloud spending.
Standard Edition
The Standard Edition is ideal for ad-hoc, development, and test workloads. It’s the most affordable option and provides access to the core BigQuery features, such as high-performance analytics, serverless computing, and machine learning capabilities. Use cases for this edition include small to medium-sized businesses that need to analyze data but don’t require advanced features.
Advantages:
- Cost-effective for smaller workloads
- Provides access to core BigQuery features
- Ideal for ad-hoc, development, and test workloads
Enterprise Edition
The Enterprise Edition includes advanced security and governance features, making it ideal for customers with more complex regulatory requirements. It also includes advanced data management and machine learning capabilities. Use cases for this edition include large enterprises with sensitive data that require robust security and governance capabilities.
Advantages:
- Advanced security and governance features
- Advanced data management and machine learning capabilities
- Ideal for large enterprises with sensitive data
Enterprise Plus Edition
The Enterprise Plus Edition includes all the features of the previous editions, along with additional capabilities designed for mission-critical workloads. Use cases for this edition include customers with mission-critical workloads that require high uptime, availability, and recovery requirements.
Advantages:
- Includes all features of previous editions
- Advanced support options, including 24/7 support and a dedicated technical account manager
- Ideal for customers with mission-critical workloads
BigQuery Autoscaler
BigQuery Autoscaler is a powerful new feature that manages compute capacity for users in an efficient and cost-effective manner. With the Autoscaler, users can set up maximum and optional baseline compute capacity, and let BigQuery handle the provisioning and optimization of compute capacity based on usage, without any manual intervention on their part. This results in sufficient capacity while reducing management overhead and underutilized capacity.
One of the main benefits of using the Autoscaler is that it allows users to pay only for what they use, unlike alternative VM-based solutions that charge for a full warehouse with pre-provisioned, fixed capacity. BigQuery Autoscaler leverages the power of a serverless architecture to provision additional capacity in increments of slots with per-minute billing.
It is especially useful when combined with autoscaling reservations, which automatically scale reservations in line with workload demands. As workload increases, BigQuery dynamically adjusts slots so that users only pay for what they use.
- Autoscaling reservations are only available with BigQuery editions.
Use Case Example
Let’s consider a hypothetical use case for a retail company that has a BigQuery data warehouse to store sales and customer data. During the holiday season, the company expects a significant increase in traffic and sales data, resulting in a surge in BigQuery queries.
Assume that the company’s baseline capacity is set at 1,000 slots, and they anticipate a surge of queries that could potentially require up to 3,000 slots. With BigQuery Autoscaler, the company can set the maximum capacity at 3,000 slots, allowing for increased compute capacity during peak times without any manual intervention.
During the holiday season, the company’s workload increases, and the Autoscaler dynamically adjusts the number of slots based on usage. Suppose the actual number of slots required for the queries is 2,500. In that case, the Autoscaler will provision the additional 1,500 slots required to meet the peak demand and charge the company only for the actual slots used at a per-minute rate.
- Without the Autoscaler, the company would need to manually increase the capacity to handle the surge in traffic, resulting in increased management overhead and underutilized capacity outside of peak periods
- GCP’s newest update, BigQuery Studio (designed for big data management in organizations using AI, ML, and LLM) is an all-in-one analytics dashboard that streamlines end-to-end analytics workflows, from data ingestion and transformation to sophisticated predictive analysis.
BigQuery Compressed Storage
BigQuery compressed storage is a feature that allows users to store data more cost-effectively by compressing data columns. This feature is particularly useful for organizations that handle large volumes of data and are looking for ways to reduce their storage costs.
The compressed storage billing model enables customers to manage complexity across all data types while keeping costs low. This billing model uses physical bytes to bill users for the storage they use, and this includes time travel storage. The time travel window can be configured to balance storage costs with data retention needs.
- When you create a dataset in BigQuery using SQL or the BigQuery API, you can choose to use physical bytes for billing instead of logical bytes, which is the default unit of consumption. However, once you change the storage billing model to use physical bytes, you cannot change it back to using logical bytes.
BigQuery New Storage Pricing
- BigQuery storage pricing is based on the amount of data stored in tables, and it is prorated per MB, per second.
- Active storage includes any table or table partition that has been modified in the last 90 days.
- Long-term storage includes any table or table partition that has not been modified for 90 consecutive days, and the price of storage for that table drops by approximately 50%.
- Each partition of a partitioned table is considered separately for long-term storage pricing. If a partition hasn’t been modified in the last 90 days, the data in that partition is considered long-term storage and is charged at the discounted price. However, if the table is edited, the price reverts back to the regular storage pricing, and the 90-day timer starts counting from zero.
How to Prepare for BigQuery Pricing Changes
As BigQuery transitions to a new pricing model, organizations using the service must prepare to adapt to the changes to avoid any negative impact on their data analysis and management workflows. Starting July 5, 2023, BigQuery customers will no longer be able to purchase flat-rate annual, flat-rate monthly, and flex slot commitments. This means that customers already using these pricing models will need to migrate their flat and flex capacity to the right edition based on their business requirements, with options to move to edition tiers as their needs change.
In addition to this, the price of the on-demand analysis model will be increased by 25% across all regions, starting on July 5, 2023. This will impact organizations that rely heavily on on-demand usage, making it more important for them to optimize their usage and queries to minimize costs.
- Organizations need to analyze their BigQuery usage and queries to evaluate and identify areas of optimization. This can be done by looking at query logs to identify high-cost queries, inefficient joins, or long-running queries that could benefit from optimization. Once these areas have been identified, organizations can migrate to the appropriate BigQuery edition that best suits their needs.
- To make the most of the new features introduced by BigQuery, such as Autoscaler and Compressed Storage, organizations need to review their workflows and determine the best way to use data. Autoscaler can help organizations save costs by automatically adjusting the number of virtual machines needed to run queries based on workload demand, while Compressed Storage can help organizations save on storage costs by compressing data stored in BigQuery.
- To further optimize BigQuery costs, organizations can follow best practices such as using partitioning to reduce query costs, optimizing table schema and reducing the amount of data scanned during queries. Regular monitoring of BigQuery resources is also recommended to ensure optimal usage and cost efficiency.
Conclusion
By now, you should have a better understanding of how to get the most out of your BigQuery investment, especially with the upcoming pricing changes.
Remember, as an organization, it’s important to analyze your usage and queries to identify areas of optimization, migrate to BigQuery editions that suit your business needs, and leverage new features like Autoscaler and Compressed Storage. By doing so, you can ensure that you’re using your BigQuery resources effectively and efficiently.
Don’t forget to follow FinOps practices and constantly monitor your BigQuery resources to ensure cost efficiency. By doing this, you’ll be well on your way to maximizing the value of your cloud investment.
