Tom: Hello and welcome back to this next TomCast from Iron Bow Technologies. We are a tactical cybersecurity as a service organization dedicated to helping businesses protect their data, their assets, and their endpoints. Today we’re going to discuss a portion of critical infrastructure, namely water and wastewater treatment facilities. Joining us in this discussion is Iron Bow’s own Dr. Barry Wood. Welcome, Barry.

Barry: Thank you. It’s good to be here.

Tom: I have a few questions to ask to help the listeners understand why this is an important topic, why this is considered critical infrastructure, and why they should be paying attention to these facilities and services outside of daily use and convenience. So Barry, to start off, what’s the role of wastewater systems in critical infrastructure?

Barry: Well, in critical infrastructure, we refer to the water and wastewater sector or WWS. It’s two sides of the water coin, as it were. The water sector brings clean water to you for drinking and bathing and what have you, and not just to you. I mean hospitals, schools, businesses, power plants, you name it, anybody who needs water, water sector brings it to them, and the wastewater sector takes dirty water away from you. Again, not just you like your kitchen sink outflow and your toilet flush and so on, but also things like industrial and agricultural wastewater because they generate a lot and it doesn’t just disappear. It has to be controlled and cleaned and sent somewhere, and eventually it’s treated and rereleased into the environment for irrigation and so on. So that’s what these two sectors do, and if you think about how vital water is to human survival, let alone comfort, you can’t go, what is it, two days or something, three days without water before you shrivel up and die. So the function of this industry or this sector in getting water to millions of people and then taking away the wastewater from millions of people makes you understand why it’s called critical infrastructure.

Tom: Got it. Wow. Okay. Well, what are some potential cyber attacks on the water and wastewater sector?

Barry: Well, for some years people have been talking about something called the IT, OT convergence. IT is computers, what people think of as data internet, apps, so on, so forth. And OT is operational technology, so this is machines that actually do things. They push, pull, pump, twist, cut, burn, et cetera. So this is how we affect our will on the natural world, so to speak. Now, this convergence of IT and OT, so bringing digitization into operations has been an incredible boon to industry generally and to water and wastewater systems in particular. It makes for a fantastic efficiency. You can have better service with lower prices, cleaner water for less effort and so on and so on. But this does give rise to vulnerabilities As operations OT systems are integrated with IT, you start to get some of the familiar cyber weaknesses that hopefully more and more people are familiar with.

For example, it’s great to have IT and OT become integrated, but often people don’t properly separate their IT networks from their OT networks. In the same way that the developers of your software should not have access to the salary information of the rest of the company. You want to keep that segmented, so you do not want just anybody’s IT machine on their desk talking to your controllers and pumps and valves and so forth. Also, one of the great things about the IT OT convergence is making remote access possible, because if you think about it, a lot of water facilities are out in kind of the middle of nowhere serving people in very, very sparsely populated areas. Sometimes very dry areas, maintain something, it’s difficult to fly or drive out there or something and do it with your own two hands. So it’s been great to be able to just log in from afar and make changes to settings and so on to set things right.

But remote access is a two-edged sword. It’s great for contractors, but it’s bad if it’s insecure and kind of anybody, whatever their intentions may be, can log into your controllers and your pumps and your valves and so forth. That’s very bad news. Related to this is the fact that sometimes people install devices and systems and for ease of access I suppose, the devices are accessible from the public internet. So you can actually just sit at your browser and enter a certain IP address and boom, you’re looking at the control panel for a certain device. Again, highly convenient but really bad if the person using the browser has bad intentions. Another thing related to that is that when people buy and install devices, like something called a programmable logic controller, which is vital for keeping systems running and taking in data and making any changes that are necessary based on it, they’re wonderful things, but often they have default credentials.

So when you buy it and take it out of the box, it’s got a password built into it, and you really should change this password when you install the thing. But when people don’t, these passwords are public knowledge. You can right now Google the name of a company or a controller or something and find a list of the default passwords. It’s not super complicated. I mean, often it’s like admin, but you do want to change that and when you don’t, and this information is available on the internet recipe for trouble. And finally, I would just say that password hygiene is everybody’s problem. Keeping passwords sufficiently complex, changing them often enough, changing them when somebody leaves. So if somebody has the keys to the castle and he or she quits or gets fired, you don’t want them to take their password with them. You want to change it. If that doesn’t happen again, recipe for trouble. So these are kind of some of the cybersecurity issues that are well known from IT, but when you apply them to OT, which affects the physical world, that is a whole different and more serious kettle of fish.

Tom: Yeah, good grief. That’s a lot to absorb. What might the damage be from a successful cyber attack on the wastewater and water sector?

Barry: Well, the possible scenarios kind of run a gamut from relatively benign to scary. As relatively benign I would classify something like if hackers get into a water or usually a water facility and muck around with the online bill payment app. So you have trouble paying your water bill online. That’s bad. It frustrates people. It makes the company look bad, but as hacks go, it’s relatively benign. Nobody got hurt. Less benign is when hackers do something like that, but then they steal and release people’s personal data. It’s a water company. They’ve got your name, they’ve got your address, they’ve got your credit card information and whatnot. If somebody takes that and then puts it up on a leak site or what have you, that’s bad. That’s a big breach. These are in the news these days. Worse than that, and now we start getting into the operations side of things is if hackers attacked the IT of a water company, but as I mentioned before, it’s not properly segmented from operations, then the IT attack can possibly jump the fence and cause trouble in the machinery.

If you remember the Colonial Pipeline event from a couple of years ago now, I think it was, what was it, late 2021 something. Colonial Pipeline, the company was hit with a ransomware attack and they were afraid that it would jump the fence and interfere with operations, so they shut down the pipeline. It’s not that the hackers shut it down. It’s that the company as a preemptive strike so to speak, shut down the pipeline. So the attack wouldn’t affect operation systems. Worse than that is when there’s an environmental impact. I mentioned that wastewater treatment plants take the effluent from your house and factories and farms and so forth, and they clean it and they send it back into the environment, either into rivers or as irrigation water or what have you. Well, if the water is sent where it shouldn’t go, or if worse it’s not treated and then sent where it shouldn’t go, you have the makings of a very bad environmental situation.

Worse than that, so we’re getting to the end of the spectrum now, is if a cyber attack actually cuts off people’s water supply. So you and I take it for granted that when we turn the faucet clean, drinkable water is going to come out. Imagine turning the faucet and nothing comes out. This is a possibility. A hack could conceivably shut down the operations of a water supply facility, meaning they can’t supply water. And then the far end of the spectrum, which is positively scary, but I will say fortunately, unrealistic is a water attack that poisons the citizenry. That people don’t cut off your water supply, they put chemicals or something into it that make you sick or worse when it gets to your house and your kitchen table. Now, I say that’s unrealistic because thankfully, there are sufficient safety mechanisms built into the water supply system, including literally people in white coats with test tubes that make this very, very unrealistic, but it should never be dismissed as impossible, just unrealistic. I’d say that’s the gamut of possible damage from a cyber attack.

Tom: Looking at today’s threat landscape, what are the actual cyber threats that could impact the water and wastewater sector?

Barry: Well, as it happens, everything I just mentioned has actually happened except for the last one, thank goodness. So if I go from one end of the spectrum to the other, just last month, so this is January of 2024, a very large water supplier called Veolia North American Municipal Water at a ransomware attack on their backend systems, and this hobbled their online bill payment system and was a headline because they had to come out and say, yes, we’re under cyber attack. This is why you can’t pay your bills. A similar attack happened in the UK. Company’s data was stolen by hackers and put up on a leak website. So this is the personal information I told you about names and credit card numbers and so forth. Back in 2016, hackers attacked the online payment app of a company, which has remained anonymous in the reporting, but they hacked the online payment app, probably hoping to get people’s credit card information, and they found themselves in control of valves and flow control.

Now, thankfully, they didn’t know what to do with it. If they had, it would’ve been worse, but I think it was surprising even to them that all of a sudden they got into the network and they zigged and they zagged, and all of a sudden it’s like, okay, what level do you want the water at? The first cyber attack ever on critical infrastructure was a actually wastewater attack, and this was back in 2000, a disgruntled ex-employee of a water treatment plant in Australia drove around with a PC and a radio set up and by radio manipulated the pumps and released 265,000 gallons of raw sewage into public areas. We’re talking parks, streams, and the grounds of a hotel, and it took days to clean up, and I’m sure it’s stank to high heaven, and he got two years only. Anyway, continuing down the spectrum last December, people in two Irish towns did have no water for two days.

Their water supply simply stopped because hackers had defaced and compromised a device in the pumping system that supplied water to these two towns. Now, it was fixed, and it’s extremely inconvenient not to have your running water, but in a civilized country, you can get it from other places, I imagine. I don’t know what they did trucked in water or something, but the water was actually cut off. This is similar to an attack that got a lot of headlines here in the US in Western Pennsylvania last November. There was a probably related hack where hackers got access to a device and essentially vandalized it. So they put a politically oriented notice up on the screen instead of seeing the controls, you saw this anti-Israel propaganda. Thankfully, that system was quickly just taken offline, and they ran it manually until they could fix it. Your backup is always human beings.

Now, that gets me to the far end of the spectrum. No one has actually ever poisoned a water supply, but it’s not for lack of trying. A couple of years ago, about five years ago, again, a disgruntled ex-employee in central Kansas used his phone to shut down the cleaning and disinfecting processes at his former employer, which is the water treatment plant. Again, there are so many safety mechanisms that this was noticed and put a stop to, so nobody got sick or anything, but he tried it. I think the most famous hack in the water sector is the Oldsmar in Florida. I think almost exactly three years ago was in the headlines because apparently somebody had hacked into the system and an operator noticed his cursor moving without him controlling it, and they changed the levels of lye, sodium hydroxide in the water and cranked it up like 11,000 times or something.

Now, happily, I don’t know if everybody knows this, this was actually not a hack. This was operator error, and the reason they thought it was a hack is because they were using a remote access app. So a contractor could log in from another location and make such changes. They weren’t sure at the time that it wasn’t somebody hacking that app. In the end, thankfully, it was not an actual hack, just operator error, but it did make headlines, and I think it made people very aware of what could happen to our water if the wrong people got their hands on it. You can build on all this. I mean, construct your nightmare scenario. 10 years ago, almost a bad actor actually got access to the sluice gate of a dam in New York and heaven be thanked. The sluice gate was actually offline for maintenance at the time, but the theory is that he could have opened it and sent the dam water emphatically where it should not be.

Also, researchers in Israel have demonstrated the potential of a malicious botnet to take over. so-called Smart Irrigation Systems. These are systems that irrigate the land automatically based on data that they get from sensors about the weather. Is it raining? Is the soil saturated, so on and so on. If you hack that, you can tell the things that, oh, it’s not going to rain for days, so you better pump everything you can, or conversely, it’s pouring so turn off. And they show that you could actually drain a standard water tower in less than one hour with just about 1300 bots hacked sprinklers, I guess you would call them. So in a place with very little water, that’s really bad news. So again, back to basic principles. If you think about how important water is to our life and how integrated it’s becoming the broader IT world, you want to keep the bad guys as far away from it as possible.

Tom: Good grief. You start thinking doomsday scenarios and such. Final question, what can people do about it? What can people do to help protect the water and wastewater sector?

Barry: Well, that’s a good question. It’s actually difficult, or let’s say challenging to secure the water and wastewater sector for a couple reasons. Mainly it’s financial. From this perspective, in the United States, there are over 50,000 drinking water plants and over 16,000 wastewater treatment plants that are municipally own, meaning their operating revenue comes from local taxes, right? It’s not a private company that can just pass on any added cybersecurity costs, let’s say to the consumer, if they want more money, they have to raise your taxes, and that is a political adventure for anybody. So they have limited budgets. A limited budgets also mean limited personnel. I mean, at some of these hacks, like the one in Western Pennsylvania recently, I think there were two engineers at the plant, and neither of them is going to know cybersecurity from a hole in the ground. Just like you don’t know how to run a water plant.

So if you have few personnel and usually very limited cybersecurity knowledge in the sector, that’s a challenge to be overcome. You’ve often got very old equipment. In operations, technology generally, you get a lot of things that were installed 20 years ago because they were going to last, right? So there’s no need to patch them or fix them or even look at them because if it ain’t broke, don’t fix it. Well in IT, how often do you patch your computer or an app once a month more. These are things that have been running for literally 20 or 30 years, so they’re often extremely old and even overlooked when they’ll do an asset inventory to see what kind of things are on their system. The 20-year-old device doesn’t necessarily show up, but if it becomes accessible, then that’s going to be trouble. Now, all these things are what make water and wastewater such an appetizing thing for bad actors.

It’s not that they have a particular fixation on water. It’s that they know, like the old saying, why do you rob banks? That’s where the money is. Why do you hack water plants? Because their cybersecurity is so poor. My general advice basically to everybody is have a plan. When you read about these hacks, like, oh, a device was compromised or ransomware took down the online bill payment system, ask yourself, what if that was me or us, or our company? What would we have done in that case? So know what could happen and have at least some idea of what you would do in that situation. Also, have basic cyber hygiene in place. If a device has a default password coming out of the box, change it. Use multifactor authentication. Okay, so contractors or whoever, if they’re going to log in and fix your pump or what have you, they should need to use two forms of authentication. Would up firewalls to keep out things like brute force attacks or known malicious IPs, because we do know IPs from which these attacks occur.

Properly. segment your system, your IT should not talk to your OT more than necessary, and for goodness sake, don’t open it to the internet. I should not be able to sit at my kitchen table and turn your pumps on and off. That’s a very basic architecture rule. And finally, keep an eye on things. Monitor your systems, monitor the vulnerability announcements that come out. CISA, the government agent, what is it? Cybersecurity and infrastructure security agency. They’re very good about being a clearing house for announcements and warnings about hacks or vulnerabilities in various systems. You can sign up for free updates, and there’s often recommendations either from CISA or from the company that made the thing on how to address a vulnerability. Uncle Sam does try to help. I mean, CISA, as I mentioned, has these announcements. Both they and the EPA offer some free services. If you ask them to, they will scan your system. They offer some advice, but for true security, you need true professionals. So nobody wants to outsource cybersecurity to the government, of course, but we need it. So I would recommend to, I mean, anybody in critical infrastructure to look into how to have somebody who knows what they’re doing, take care of your cybersecurity, monitor your systems, make sure everything is set up correctly, have an incident response plan ready to go in case, the red flag does go up and so on.

Tom: So you’re talking, engage with your possible managed security services provider.

Barry: Absolutely

Tom: Outstanding. That’s great advice. Well, if you have any questions surrounding water, wastewater sector, or critical infrastructure, reach out to us over here at Iron Bow Technologies. Our highly skilled team of professionals can help answer whatever questions or concerns you may have. We’re also here to help any way we can to ensure your organization remains as secure as possible.

We here at Iron Bow, thank you for taking the time to listen to this TomCast. For more information on various cybersecurity tips, head on over to our website and check out more TomCast. Those are located over on www.guardsight.com/tomcast. Or if you would like more information on what Iron Bow can do for you, head on over to www.ironbow.com and contact us. There are several free cybersecurity tools out there that can help you improve your overall security posture. We’d love to hear from you. Thanks!