Developing a Real Time Log Analyzer

I had a requirement to build a log analyzer to look for trends and performance-ish issues, which would then be persisted to a database, from which we could query and build reports, etc.

The requirements were for us to get as close to “real time” as possible, without impacting the current system much at all. Perfect requirement, right?

It turned out we got stuck on the implementation – how were we going to make this analysis available, through some ETL job, but without causing performance impacts? And also be very nearly real-time? (The actual requirements were for 10-15 minute increments.)

We had some discussions on what was possible and it boiled down to “what are the requirements?” I was obviously a little annoyed when I restated the minute-interval again.

But what was the goal? Hm, there must be something here, because of the questioning.

What is the user going to use this particular information for? Reporting, seeing spikes in activity, performance, etc.

Ah, but we already have tools (alerts) for that sort of thing. And we have a Production Support team whose existence is to stop something in Production once it occurs – to find the issue, find a way to fix it and then fix it.

So, the question again – What is the information going to be used for?

Ideally, we would do real-time, but it turned out the solution we were developing wouldn’t (ever) be able to do that. And, there are other tools to do the type of granular information we were looking at. No, our tool wouldn’t – couldn’t – do real-time. Or even near-real-time. The ability to react to those spikes or valleys isn’t something that the tool should be designed for. And at the point you are wanting real-time (or 10 minutes from it), you are already late. It’s already old information.

A hard reality to face. What makes more sense is a one-hour cron to capture every hour and graph. Anything less than that time interval is unrealistic and nearly useless vs the cost to implement.

I think as the Analyst, I should have caught this and steered our stakeholders away from this approach – but I had the “you can do anything with technology” mindset, which proved to harm more than help here.

But, lesson learned for me. Sometimes what you want to do, you can’t have. But that’s called making decisions.