I remember back when mobile devices started to gain momentum and popularity.

While I was excited for a way to stay in touch with friends and family, I was far less excited about limits being placed on call length minutes and the number of text messages I could utilize … before being forced to pay more.

Believe it or not, the #646 (#MIN) and #674 (#MSG) contact entries were still lingering in my address book until a recent clean-up effort. At one time, those numbers provided a handy mechanism to determine how close I was to hitting the monthly limits enforced by my service provider.

Along some very similar lines, I recently found myself in an interesting position as a software engineer – figuring out how to log less to avoid exceeding log ingestion limits set by our observability platform provider.

I began to wonder how much longer this paradigm was going to last.

The Toil of Evaluating Logs for Ingestion

I remember the first time my project team was contacted because log ingestion thresholds were exceeding the expected limit with our observability partner. A collection of new RESTful services had recently been deployed in order to replace an aging monolith.

From a supportability perspective, our team had made a conscious effort to provide the production support team with a great deal of logging – in the event the services did not perform as expected. There were more edge cases than there were regression test coverage, so we were expecting alternative flows to trigger results that would require additional debugging if they did not process as expected. Like most cases, the project had aggressive deadlines that could not be missed.

When we were instructed to “log less” an unplanned effort became our priority. The problem was, we weren’t 100% certain how best to proceed. We didn’t know what components were in a better state of validation (to have their logs reduced), and we weren’t exactly sure how much logging we would need to remove to no longer exceed the threshold.

To our team, this effort was a great example of what has become known as toil:

“Toil is the kind of work that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.” – Eric Harvieux (Google Site Reliability Engineering)

Every minute our team spent on reducing the amount of logs ingested into the observability platform came at the expense of delivering fewer features and functionality for our services. After all, this was our first of many planned releases.

Seeking a “Log Whatever You Feel Necessary” Approach

What our team really needed was a scenario where our observability partner was fully invested in the success of our project. In this case, it would translate to a “log whatever you feel necessary” approach.

Those who have walked this path before will likely be thinking “this is where JV has finally lost his mind.” Stay with me here as I think I am on to something big.

Unfortunately, the current expectation is that the observability platform can place limits on the amount of logs that can be ingested. The sad part of this approach is that, in doing so, observability platforms put their needs ahead of their customers – who are relying on and paying for their services.

This is really no different from a time when I relied on the #MIN and #MSG contacts in my phone to make sure I lived within the limits placed on me by my mobile service provider. Eventually, my mobile carrier removed those limits, allowing me to use their services in a manner that made me successful.

The bottom line here is that consumers leveraging observability platforms should be able to ingest whatever they feel is important to support their customers, products, and services. It’s up to the observability platforms to accommodate the associated challenges as customers desire to ingest more.

This is just like how we engineer our services in a demand-driven world. I cannot imagine telling my customer, “Sorry, but you’ve given us too much to process this month.”

Pay For Your Demand – Not Ingestion

The better approach here is the concept of paying for insights and not limiting the actual log ingestion. After all, this is 2024 – a time when we all should be used to handling massive quantities of data.

The “pay for your demand – not ingestion” concept has been considered a “miss” in the observability industry… until recently, when I read that Sumo Logic has disrupted the DevSecOps world by removing limits on log ingestion. This market-disruptor approach embraces the concept of “log whatever you feel necessary” with a north star focused on eliminating silos of log data that were either disabled or skipped due to ingestion thresholds.

Once ingested, AI/ML algorithms help identify and diagnose issues – even before they surface as incidents and service interruptions. Sumo Logic is taking on the burden of supporting additional data because they realize that customers are willing to pay a fair price for the insights gained from their approach.

So what does this new strategy to observability cost expectations look like?

It can be difficult to pinpoint exactly, but as an example, if your small-to-medium organization is producing an average of 25 MB of log data for ingestion per hour, this could translate into an immediate 10-20% savings (using Sumo Logic’s price estimator) on your observability bill.

In taking this approach, every single log is available in a custom-built platform that scales along with an entity’s observability growth. As a result, AI/ML features can draw upon this information instantly to help diagnose problems – even before they surface with consumers.

When I think about the project I mentioned above, I truly believe both my team and the production support team would have been able to detect anomalies faster than what we were forced to implement. Instead, we had to react to unexpected incidents that impacted the customer’s experience.

Conclusion

I was able to delete the #MIN and #MSG entries from my address book because my mobile provider eliminated those limits, providing a better experience for me, their customer.

My readers may recall that I have been focused on the following mission statement, which I feel can apply to any IT professional:

“Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” – J. Vester

In 2023, I also started thinking hard about toil and making a conscious effort to look for ways to avoid or eliminate this annoying productivity-killer.

The concept of “zero dollar ingest” has disrupted the observability market by taking a lead from the mobile service provider's playbook. By eliminating log ingestion thresholds, it puts customers in a better position to be successful with their own customers, products, and services (learn more about Sumo Logic’s project here).

From my perspective, not only does this adhere to my mission statement, it provides a toil-free solution to the problem of log ingestion, data volume, and scale.

Have a really great day!