5 Effective Cloud Logging Cost Optimization Strategies

Overview 

The rapid growth of the IT industry comes in small to large infrastructures and applications where it’s very common to get multiple bugs, errors, or crashes so to troubleshoot those issues, organizations need to identify the portion from the issues that are originating which is another challenge. Here log data comes into the picture, log data is invaluable for managing, maintaining, and troubleshooting IT systems. It allows monitoring across the system to detect particular log events and patterns in the log data. 

To enable the same thing over cloud infrastructure, Cloud Logging is the solution which is a fully managed service that allows storing, searching, analysis, monitoring, and alerting on logging data and events from Google Cloud Services. To use Cloud Logging Service on GCP, certain charges need to be paid and as the application will scale, the charges will also increase, so today in this blog we will discuss how to optimize the cost for cloud logging. 

Usecases of Cloud Logging

After understanding cloud logging let’s dive into some of its use cases in brief.

• Log Searching: Searching for log files are pretty easy. Where the data is not structured, when logs are sent to the logging service, it typically includes a lot of additional metadata or fields that provide context like, Server Name, Environment, Application Name, Location, Logging Level, Custom Fields, and many more. Being able to search by specific fields makes it much easier to find the information developers are looking for.

• Structured Logging: One of the most powerful things we can do in cloud logging is to log additional data or objects from code. Then it will be effortless to search by those custom fields.

• Log Monitoring: Now all the logs are in the same place and have powerful searching capabilities, we can monitor logs. These logs can be a wealth of information that can be used for monitoring specific application behaviors.

• Application Errors: Errors that happen in the application code are important to track. Logging is often the first line of defense when it comes to identifying application problems or issues. Be it database timeouts, null reference exceptions, or any other type of error. Cloud logging service offers reporting and alerts around all application errors.

How to use Cloud Logging

Let’s look at how you can view and query logs in Cloud Operations, starting with Logs Explorer. To access the Logs Explorer, go to the Products menu. From Logging, select Logs Explorer. Initially, this shows you all logs in your project. On the left side, you can use Log Fields to select just the logs that you care about.

The histogram in the dashboard shows you the volume of logs that match your query over the selected time Interval. If you want to zero in on a specific point in time, just drag the selectors to the time you care about and rerun the query. Logs that match your query are shown in the Query Results section. To get the full details of a log entry, select it. This will show you both the metadata and the payload. 

Logs Explorer even generates suggested queries for you based on what’s happening in your project. This is how you can use Logs Explorer to find the exact log entries you need to determine what’s happening with your service. The Logs Explorer is great at helping you find the logs that you need when you need them. But what if you’re debugging a service and want to watch the logs in real time? This is where log streaming comes in. 

In order to stream logs, you first have to find the logs that you care about. You can do that by writing a query or by using the log field selectors. 

Cloud Logging Log Based Metrics Pricing & Costs

Almost all features of Cloud Logging are free of cost, except the ingested log volume over the free allotment i.e., the First 50 GiB/project per month, and once the free volume is exhausted its prices are based on usage and its costs $0.50 per GiB of log ingested.

Cloud Logging Cost Optimization Methods

As of now, we understand how cloud logging gets billed, let’s understand some of the best practices to optimize the spending:

• Configure Log Sink to Google Cloud Storage/ BigQuery: Charges are not applied for storing logs for the default retention period. As such, you can reduce your log storage costs by carefully considering whether you need to retain logs in Cloud Logging for longer than default retention (i.e., 30 Days). 

If you do need those logs for other use cases, consider exporting them to Google Cloud Storage for archival or BigQuery for analytics. In this way, you don’t have to pay for them in Cloud Logging.

• Configure Log Exclusions: Primary way to reduce ingestion costs is log exclusions. Exclusions are configured when you set up log sinks. If you’re looking to reduce the volume of logs being ingested, you can modify the default sink to add an exclusion filter. 

For example, you may exclude:

1. HTTP 200 OK responses from request logs: For applications with HTTP 200 OK responses might not provide many insights and can produce a high volume of logs for high traffic.
2. Logs from Cloud Load Balancing: Sometimes, load balancers can produce a high volume of logs for applications. For example, use a logging filter to set up an exclusion for 90% of responses from Cloud Load Balancing.

Remember that excluded log entries won’t be available in the Logs Explorer or Cloud Debugger. Log entries that aren’t routed to at least one log bucket are also excluded from error reporting. But with user-defined, logs-based metrics, our computers can log entries in both included and excluded logs.

• Reduce Logs: One way to reduce log ingestion costs is to produce fewer logs from your applications and another way is to leverage Log Optimization tool. For example, you can carefully adjust the log levels to control log volume. This is especially useful for serverless services, where there’s no way to view excluded logs.

• Configure to send Less Data: Streaming logs from third-party applications, reduce log volumes by using the logging agent on only production instances or by configuring it to send fewer data.

Conclusion

Considering these steps for managing log volume in your application and using exclusions to balance observability and costs will definitely help you to save money. Having a good understanding of these models and services can help companies manage GCP costs and choose the best solutions for their specific needs.

So, if you’re looking to get started with Google Cloud, be sure to take advantage of the Pricing Calculator to help you make informed decisions. Click here to get started with a free demo and learn more about how Google Cloud can benefit your business.