- When it comes to the healthcare industry, developing an effective, efficient, stable, and easily scalable big data analytics infrastructure is an infinitely complicated proposition.
From vendor selection to product implementation to the inexact art of architecting smoothly-operating workflows for end-users, data-driven healthcare has historically been filled with more guesswork and hopefully crossed fingers than most IT professionals would like.
In a healthcare provider, big data flows in multiple directions. It may originate in the consult room, and travel from the patient’s mouth into the EHR and then into a centralized big data repository or other database – but it might also make a return journey from a storage center back to the clinician’s laptop.
Medical devices, facilities systems, scheduling and workforce management tools, and even parking meters may all generate similar bidirectional rivers of data. As a result, a plethora of vendors and proprietary products must be appropriately monitored and maintained.
One method of keeping tabs on the busy bits and bytes is through wire data monitoring. Wire data, which is the readable signature left by information as it moves back and forth across disparate systems, helps organizations understand how efficiently a particular system is performing.
In addition to identifying data siloes and budding breakages that may be dampening the smooth movement of big data across the network, it can provide key metrics to help pinpoint failures and devise solutions to issues that may be affecting the user experience.
At Phoenix Children’s Hospital, rapid growth over the past decade has brought a broad range of health IT initiatives under the purview of Chief Information Officer David Higginson. With the help of IT Service Professional John Vaux, the tech team at the top-ranked pediatric care center is leveraging its wire data to deliver comprehensive monitoring and management of its dozens of critical components.
In this Q&A, HealthITAnalytics.com explores how wire data is changing the way Higginson and Vaux do business with their internal staff and external business partners.
What are some of the challenges of managing such a complex IT infrastructure? Why did you decide to invest in a wire data monitoring tool?
David Higginson: Growth doesn’t happen particularly evenly. We have about 180 different vendor systems in operation at the hospital, ranging from meal service for the patients to the most sophisticated radiology and cardiology imaging products available.
A children’s hospital is a microcosm of every part of life, so we have a school, we have a cafeteria; we have a security department – just a vast array of systems. And on top of that, we have all of our infrastructure.
There are a lot of sites, and a lot of pieces of clinical equipment that attach to the network and produce data that have to be monitored and often can go rogue. Everyone's got a niche product, and everything has to be monitored, and sometimes it’s hard to develop a really controlled growth pattern for these things.
All those systems can generate a huge amount of data. But often, any tools that are available for monitoring are very narrowly focused on a specific product. That may be appropriate for a network engineer concerned about the performance of a particular piece of technology, but it doesn’t give us the big picture.
When it comes to growing an IT department and supporting the hospital, we can’t look at things inside of siloes. If we’re seeing poor performance in application, we need to understand all the elements, whether that's storage, memory, compute, network, SQL service, tuning, or whatever it may be.
We use ExtraHop, which has really been unique in allowing us to see all of that in one place at the same time. That means we don’t have to keep running around to different groups, which all point at each other saying, “Oh, I think it must be your fault, because my side is working fine.’
John Vaux: It really helps to hold our vendors and our application developers and our database developers accountable for what their systems are actually doing. We’ve been building a large data warehouse, and we’re putting all of our data from all of our applications into this one location, and then using that set-up to operate on it.
The problem is that when you have so much information in one place, it’s hard to figure out what’s happening when something goes even slightly wrong. Traditionally, you would use an SQL tool called Profiler, but unfortunately, that causes the database to run even slower when you’re using it.
As we aggregate more and more data, and it gets harder and harder to see what’s actually occurring, we need a tool that won’t cause us to take a performance hit when using it.
What are the benefits of adding wire data monitoring to your analytics strategy?
David Higginson: So, one of the challenges with the typical approach is that you generally wait until you've got a problem, then you turn on the monitoring. And then, just like when you take your car to the dealership, the problem never reproduces itself for the three hours you spend watching it.
So, the advantage of using a wire data monitoring tool is that it's always on. So, whenever something happens, we can immediately go back to whatever timeframe we need to and start looking at where the issue lies.
You can take the network data, the storage data, the compute data, and maybe the actual things happening inside the database and time-sync them together, so that you can really see how all five or six factors that affect the environment are all operating at the same time. That’s much more efficient than looking in six different tools and trying to piece that picture together.
It’s also important from the management side. After managing developers for years, I know that if you tell them something’s not working with their code, they’ll spend most of the time telling you why you’re wrong.
Taking a new approach to monitoring cuts out a lot of that conversation. We know exactly what went wrong, when it happened, and why it happened. Most of the time, we can get pretty close to what the root cause is. That means we can stop wasting time pointing fingers and start fixing the problem.
Additionally, our core EMR is actually hosted by Allscripts in North Carolina. For most of time, that's a bit of a black box for us. But when our users call up and tell us that the EMR is running slow, it’s not sufficient for us to say, “Well, everything looks good on our side.” That doesn’t solve the problem.
But now, because of our ability to really tap into the wire, we can see metrics originating from the hosted site without having to ask for them every time. So the ability to deep-dive not just into the network layer, but into multiple layers behind that and see into applications like Citrix – even when we’re not hosting them on our own network – has given us an extra set of tools.
Do your business partners appreciate the additional insight? How has it changed the way you interact with vendors?
David Higginson: We have some incredible vendors here who are very, very supportive…and then we have some vendors where we might only be a small client of theirs, or they feel like we should take the primary responsibility when we experience an issue. We don’t have to play around with pointing at the vendors and the vendors pointing at us, and none of us getting anywhere.
So we certainly get into situations where we'll try really hard on both sides for two or three days to resolve a problem, but then we both throw our hands up in the air and admit that we really don’t know what’s going on. That’s when it could start to get acrimonious, because then, obviously, the problem has to be escalated within the organization.
Proactively monitoring the data stops us from having to get to that point with a vendor. We can work in a more equal way. If they are saying that the issue is on our side, we can verify that and send them the logs so they can see it. It shortens that cycle and maybe even stops us getting into that adversarial relationship, because we can just get some exposure to it sooner.
John Vaux: We have a lot of those kind of conversations, unfortunately. It’s so much easier and more productive just to show someone the data. Here it is. Such-and-such took this long to happen; this particular process failed. It’s so much easier for everyone.
None of it is ever malicious, and no one gets adversarial on purpose. But it’s the nature of IT. I can’t or adversarial on purpose. I think it's just the nature of IT. A lot of the time, you’re just tinkering with things to see if this or that can fix the problem. You’re throwing stuff at the wall and hoping that something sticks. But when you can actually look into the wire, you can see what’s happening much more clearly. It’s changing the nature of the conversation.
How does this approach help you to be more proactive when it comes to addressing potential problems?
David Higginson: About ninety percent of the things we rectify and improve are not things that anyone is actually calling about. Which may sound crazy. Why is it important if no one is complaining? But that’s such a reactive stance, and we want to be so much more proactive than that. Each of those issues could easily become a problem later on. Catching them ahead of time is exactly what we want to be able to do.
For example, we may have a configuration on a network switch which has been working totally fine. But now someone has tweaked something and accidentally started a problem that typically would not have shown itself very quickly. When it does show itself, it’s often in a much harder way to diagnose. So being proactive really reduces the confusion and the effort that goes into maintaining such a complex series of systems.
What is your advice for other providers who might want to start taking a deeper dive into the way their data systems are behaving?
David Higginson: My first concern, when we started this, was that I wanted to be 100 percent sure it wasn’t going to negatively impact our systems, because my experience in the past has been that any kind of monitoring tool generally slows everything down. Obviously, if you’re already dealing with problems, that the last thing you want to do.
So the first question to address is that you have to be very sure, from a technical perspective, that you are not implementing anything that is going to adversely impact your existing environment.
The second challenge when selecting a tool is making sure that you are going to get a return on your investment. From our perspective, the only purpose of buying this system was to achieve ROI. We’re not interested in this as a hobby. We went into it with a very hard ROI goal that we were trying to accomplish, and we worked on finding the right tool at the right price so that we could attain that.
Fundamentally, the real test of whether or not you made a good investment is if patients are going to be treated better because of this. As a CIO, I’m not interested in something that is going to give me a pretty graph, but that’s it. You need to be able to see results.
John Vaux: Make sure that you’re not just buying shelf-ware. There’s no point in purchasing a monitoring tool that doesn’t monitor anything. You have to actually use your products – and be sure that everyone knows you’re using them. When we do our quarterly reports, everybody knows that David gets that data and reads the information. All the engineers down the line know that the CIO is looking at this, and that they have to do the work they said they were going to do. It’s about accountability.