Why is the industry embracing BACnet/SC? Is it a good idea?
What are the ramifications of rolling out a new protocol stack to the market?
In this episode of the Smart Buildings Academy Podcast, I discuss my thoughts on the BACnet/SC protocol and I discuss why it may or may not be a solution for your business.
Click here to download or listen to this episode now.
Resources mentioned in this episode
Show notes
Phil Zito 0:00
This is the smart buildings Academy podcast with Phil Zito Episode 238. Hey folks, Phil Zito here and welcome to Episode 238. In Episode 238 of the smart buildings Academy podcast, I'm going to be talking about why I believe BACnet secure Connect is a horrible idea. Now I want to add a little bit of background. So you all understand kind of where I come from, because I'm going to say a lot of things here. And granted, these are my opinion, you don't have to agree with me. You know what, it'd be totally fine. If you don't agree with me, I would love to hear your counter opinions. To my perspective. Now, my perspective comes from a pretty wide background. Prior to starting smart buildings Academy, I was running the integration program at Johnson Controls, I would build out BACnet stocks, I would build out API's, I worked with a slew of different data sets across multiple different buildings, systems, specialty systems, and business systems. I also have a master's degree in cybersecurity and Information Systems. I've designed networks, I've held Cisco certifications, I've held the CISSP certification. I've done a lot in regards to technology. And I've been involved in white box and black box testing of control systems and other applications for customers back in my time at Johnson Controls. Now I say all this not to toot my own horn, but rather to give you a perspective of my pedigree and background when I bring up these issues. Because at the end of the day, BACnet SC is not a bad solution. architecturally how it's created, the technologies they chose to use in it are not bad, I'm going to argue that they're not necessary. And they actually are going to complicate things further, rather than simplifying things. They are going to add potential barriers with it. And they are going to most importantly, add significant project and material costs to the industry. And we're actually moving away from where we should be moving as an industry, which is using more open data focused models that are flexible, and are decoupled from the protocol. I'll talk about that towards the end of the episode. So what is BACnet? Sc? You've probably seen a lot of the marketing from the OEMs coming out talking about BACnet secure Connect, it's an answer to three problems. One, it's an answer to the back net BBM D routing issue. It's an answer to the back net clear text issue. And it's a answer to the BACnet device, authentication and verification issue. And I'll talk through that once again. Later in the episode. Now BACnet SC uses WebSockets which are similar to HTTP, but not so WebSockets are bi directional communication. Whereas if you look at HTTP it, it uses post and get messages. So it's unidirectional. And also WebSockets tend to handle data flow, streaming data flow better and bi directional data flow better. That's why you see them used in like a lot of real time chat applications. in video games and a lot of streaming video applications, you'll see WebSockets in lieu of HTTP API's being used. It also uses TLS 1.3, which is Transport Layer Security. It basically replaced SSL after SSL got some issues, essentially, each device is going to have a certificate and that certificate is going to validate that the device is who the device says it is. And it's going to enable encryption between endpoints. So what basically happens is it uses a hub spoke methodology. There are back net hubs and are the SC hubs as they call them and SC hubs in April the devices to communicate to the hub and authenticate to the hub. And the devices can also do peer to peer where they authenticate using their certificate checking. Like I mentioned, you can have multiple hubs and the hubs see one of the MIS understandings of BACnet SC Is it still has broadcasts, you know the who is I am is still broadcasting
Phil Zito 4:57
but those messages are hanging handled through hubs, which are going and sending the information. There's much more technical aspects to it. But you're not crossing subnet boundaries. And it kind of decouples you from the physical isolation that is BBM, DS, and all of the fun stuff that causes issues with BBB. So you may be sitting here at this point being like, Phil, you have really not convinced me at all, that BACnet SC is a bad idea actually. Sounds like BACnet SC is a great idea. I mean, after all, it's going to compensate for the fact that we've utterly failed in training our bs people around it, it's going to solve the BACnet routing issue, it's going to solve the clear text issue, it's going to really help you conform with certain government security requirements. Actually, this is one of the biggest issues that drove back net se was people not being able to get on sites because back nets clear text, and they were saying, Hey, we can't have this clear text solution on our site, you need to go and have a more secure version of your protocol. In order to be on our site. It's going to solve broadcasts storms from poorly designed BBM DS, it's gonna fix the protocol address translation issues, and BBM D location issues. But is it really the best solution? Do we really need fixes for all of these issues? So first, we need to understand what are cybersecurity controls? I mean, what is a cybersecurity control? There's three types of controls, there are administrative, physical, and technical controls. If we look at building automation systems, the majority of the issues that we experience, and that they reference in BACnet, SC are on the technical and the physical aspects of the controls. So from a technical perspective, yes, back net is clear text. And in a certain context, clear tax can be very bad. in different contexts, clear text can be perfectly fine. And we've forgotten or maybe we never knew, as an industry, the number one rule on cybersecurity controls. And that is that you apply the level of controls equal to the level of acceptable risk. That's like the first thing you learn when you're getting an education in cybersecurity is threat analysis, risk profiling, and then going and assigning a value to risk based on the probability of occurrence and the impact to business continuity from that risk. And then once you've determined the actual cost of the risk, you then say, Okay, I'm willing to spend this much to mitigate the risk. Now, in a D od application in a tier four data center, yeah, maybe they do have a very, very low level of risk tolerance, they want the least amount of risk as possible. That being said, What is the majority of the assets that we are dealing with out in the built environment, office space K through 12, some university campuses, some corporate campuses, most of which are segmented from the greater it network. Now, let's just take commercial real estate and K through 12, which are the majority of the assets we deal with? If you look at those, what is the level of cybersecurity risk tolerance at those buildings? If you say none, then you're fooling yourself. those buildings absolutely have a very high acceptance of risk for most of those buildings. Because the reality is the cybersecurity risk of impacting business operations of that and the likelihood of someone attacking that system is relatively low. So because of that the amount of cost both in implementation and support for a cybersecurity control should be low. But as we'll see with BACnet, SC and the things that requires in the support it requires that's not the case.
Phil Zito 9:34
Also, we are using a protocol to solve technical security issues. Think about that for a second. We are implementing BACnet SC as a stack of protocols. And we are going and mixing all this up and yes, BACnet is an open protocol, you can go purchase the standard and you can make your own BACnet stack. But once that stack is implemented inside someone's firmware, your visibility to that stack becomes limited. So how did they implement OAuth? How did they implement WebSockets? How did they implement TLS? What if a new version of TLS comes up, and now you've got a network that because it was perceived secure, is now writing across both public and private networks, whereas before it was isolated, and we could have used different technology sets that we would have had full visibility to, that we could rapidly update in case of a security vulnerability. The thing is, whenever you bake multiple security technologies into a single protocol into a single stack of protocols, as the case of SC, you are increasing the likelihood that one of those protocols is going to be vulnerable, and you're going to give this false perception of cybersecurity. Now, that being said, when you are analyzing all of this, if you were to use some of the solutions that I'll provide later in this episode, if one of those solutions was vulnerable, you would be able to deal with that directly. It would be much easier to check that vulnerability. Now, I know some folks are gonna argue, well, a lot of folks thought solar winds was pretty secure. But it turned out it wasn't. Yes, you're absolutely right. But imagine if you had solar winds as part of a stack, and it was all built together. And in order to replace solar winds, it wasn't a matter of just updating solar winds, you had to update the entire firmware. Now, what kind of issue that you have, I hope, what you're seeing what that explanation is, by having these separated out, you can go and throttle your different levels of security, you can be aligned with customers, IT security stacks and applications. And then fit it security that they want is higher than what SC could have provided great. If it's lower, that's fine, too. But you aren't putting all of this stuff into a mixed bag, and then relying on firmware updates, and things of that nature in order to go and patch security vulnerabilities, which are coming out almost on a daily basis. But one of the bigger issues I see is that we're trying to use a protocol to solve poor design issues, poor flow regulation issues. And poor five sec issues physical security, one of the things you'll often hear folks say is if I could go into the building, and I got access to your BACnet Network, I now have access to your network. And I could do all sorts of bad things. And the issue should not be that BACnet is insecure, and that they got access to the network. The issue should be that they are physically in the building, connecting to a network and your five sec, completely failed five SEC is physical security. There's this concept of the onion, the security onion, and so you have perimeter five sec, you have entryway, five sec, and then you have restricted areas. And you know, it goes further and further. The fact that they're able to gain access physically to these assets shows multiple layers of physical security failure. And remedying that with a protocol is foolish, because at the end of the day, yeah, you may have a secure protocol. But if I can get physical access to your machine, I can route your machine and still bypass your security protocols. And so saying that this protocol is going to solve five sec issues is going to once again create a false sense of security. Also, we talked about solving poor design issues. Oftentimes the BBM D and foreign device registration is referenced as really difficult. And we have broadcast storms, we have all these issues. The issue is that there was no
Phil Zito 14:05
strategy put into establishing BBM DS and bdts BBM, DS and bdts have been very clearly defined how to go and deploy them has been very clearly defined. The fact that a campus chose not to have a strategy does not necessitate a technical solution. it necessitates a proper design strategy. That's like going and saying every building is named differently. And all the points are named differently. And now we can't use a data taxonomy and analytics because of that. So we need to go and create this new protocol and overlay everything. No, you should get in front of the fact that you have all of these disparate namings and you should go and implement a standard. Now I know I just said implement a standard and some of yours were in well, SC is a standard. So we should implement that. No, that's the thing. There's gonna be some cost that comes with that I'm going to describe in just a second, that is going to be why it's just much more effective for us to follow proper design with BACnet IP as it stands right now. Now poor flow regulation issues, this is another argument that you'll get from folks, which is that BACnet IP, and broadcast storms aren't such a big deal, until they start to hit the field controllers, which cannot tolerate that much data flow. Okay. But that's why you set up your supervisory devices properly. That's why you do flow regulation on your supervisory devices. And you control the flow rates. That's why you don't do you know, 100,000 plus baud on your field trunk. So you go and you use the 9600, or 38.4. That is specified by the mstp standard. And by implementing that you're effectively controlling data flow. Now, granted, Could your networks get bogged down by poor broadcasts or too many broadcasts? Yes, absolutely. But once again, that's poor BBM D design and BDT design that's putting too many IPS on your bdts. And expecting that to have proper flow control. So what are some viable our opportunities are viable alternatives to the solution, right off the bat VPN,
Phil Zito 16:19
if you're really concerned about secure messaging, end to end VPN, IPsec. VPN is perfectly fine, implement them. And you've essentially created a secure tunnel between the devices. But Phil, we can go and still put our networks or we can put a device on the network. Once again, that's a five sec issue. And then, you know, if you have your VPN, you're going to have secure communication between devices, you're going to get encryption. And before I go any further, I want to talk about this concept called the CIA triad confidentiality, integrity and availability. confidentiality, we're not so much concerned about confidentiality with building automation traffic, you don't really care if someone knows our set point, in most cases, I have worked at some facilities that are concerned about that, because their temperatures are part of proprietary manufacturing processes. So they care about those environmental variables. Most people care about integrity and availability. If I command a point to 72 degrees, and it comes back at 80 degrees, that's an integrity issue, if someone could DDoS, which is a directed denial of service, right, and they take down a device by basically, you know, pinging it with a bunch of different computers or whatever. That is an availability issue, both of which can be mitigated through things like firewalls, intrusion detection, intrusion, prevention, and VPN. So all of these things right there can go and actually solve our secure communication issue, they can actually go and solve our availability and integrity issues right there. Now, once again, those costs a little bit of money, but they are, in most cases, already being utilized by an organization. If an organization has a need for BACnet SC, the likelihood of their IT organization already using some sort of secure tunnel is going to be high. Now you may say, okay, Phil, we've encrypted this data, and we've secured it using a VPN, but it can still go and access all the other data on the network. Well, that's where VLANs come into play. VLANs allow us to logically isolate traffic. So we are able to essentially create a another subnet for a traffic that cannot interact with other subnets. Now, sure, there are ways to cross the VLAN boundary. But those mainly have to do with people in properly setting up their networks. So by implementing VLANs, and VPN, we've solved a lot of the issues. Now people will say, Okay, what about the BB MD and the flow rate issue? What about the broadcasting? What about the static versus dynamic IPS, which static and dynamic IPS and DNS have been largely solved by most manufacturers at the supervisory device layer. So that being said, with these technologies, we're able to go and address many of the vulnerabilities that are caused by BACnet, IP and mstp without the added cost. So now let's talk about costs I've talked about cost quite a bit. What does this look like? Now, if you go and read the back net white paper and I spent the better part of two days reading all about BACnet sc. That's why it This podcast is a little bit delayed as I wanted to really get up to speed with the latest information and kind of dig into it. really deep. I could talk to you about it. But if you look at BACnet SC, they do have, in scenario three of their white paper, they do address using BACnet, secure Connect routers that go back to the hub device that will essentially allow secure communication between the devices. So you could implement this on legacy devices. That being said, the whole purpose of BACnet IP, and BACnet mstp traffic and the devices being insecure and being able to be locally connected to if someone accesses the network that still exists, even with sc unless it's a BACnet SC device that implements cert, certificate eating and authentication, then you're still going to have that device. So ultimately, to get to the BACnet SC level, we're going to have to update to the BACnet SC firmware. Now best case scenario, you've got modern controllers, and that's simply downloading a new firmware, and configuring the devices. Speaking of configuring the devices, you're going to have to create a sir certificate for each device, because each device has its own certificate. The reality is they say, to use certificate authorities. Reality is most of the contractors out there are going to go and self signed certificates. And they may or may not use the right version of TLS for those self signed certificates. Now granted, hopefully they have security built into that the devices that detects that and will not allow that to be a valid certificate. But who knows. That being said, for a majority of you who have assets out in the built environment, you're going to have to physically update your devices in order to support the SC firmware. So now you have to ask yourself, who benefits from this protocol. Now, I'm not going to get very, very conspiracy theory. There are definitely when I put out BACnet SC has issues changed my mind post on LinkedIn. I got some folks on there who said, you know,
Phil Zito 22:06
look at the board of sc. Of course, who's got to benefit from this. I like to be a little bit more optimistic than that. And I think that people should legitimately feel that this is a solution to the problem. However, I think that we often when we see a van when we buy a van like I bought a Honda Pilot years ago, 2000 or a Honda Odyssey years ago, 2014, I think. And I started seeing Honda Odyssey all the time. Well, if all you've known is BACnet mstp. And it's all you an IP and all you've ever worked with, then you're going to see the answer to being this problem being solved as BACnet sc. There's a constraint I like to bring to my employees, whenever they bring an idea to me and they say, hey, Phil, I want to do this, or Hey, Phil, I want to do that. And I feel like we're limiting ourselves. I say if you absolutely had to do it with this limitation, what would you do? So my question is, if we had to implement secure communications, only using BACnet IP and BACnet mstp? How would we do it? How could we implement secure communications and do that? And I think that that conversation I'm sure it was had. And I'm sure that they looked at it and evaluated that. At the same time, I will tell you, what if there was no back then what if we abandoned back net? entirely? And I'm gonna come to that in just a second. Because I know that's like a crazy extreme thought. And it's something that I think, personally, I'm biased, I think it needs to happen. I don't have a dog in the fight. I don't own any products, but I think it's holding us back. But let's back up from that constraint real quick. If we look at implementing VPN, if we look at implementing VLANs. If we look at implementing intrusion, protection, and intrusion detection, or intrusion prevention and intrusion detection, we've got a viable solution that solves most of the problems if we properly implement BACnet IP, we solve most of the bbmp and BDT issues. We're implementing a solution with sc for a small subset of customers. For those of you who have never worked for a product organization, what happens is your biggest customers influence the direction of products. So what will happen is the GSA or the D od or the US military will come along. And they'll say we've got Phipps whatever, or we're doing this standard. And in order to be certified, you know, back in the day, it used to be like ditz cap and die cap. It's it's not that anymore, but in order To be certified to be on our system, you need to meet XYZ. And so then the salespeople who handle those federal accounts, they freak out, they're like, Oh, crap, we're not gonna be able to qualify. So they go back to the product organization, and they say, you absolutely need to implement this, this, this and this, and the product organization goes, Oh, well, we can't do that. So we're going to go back to the protocol and say, we need to do this. And then you get a protocol that is being driven not by the market need, but by several large customers. Because if you look at what SC does, and you look at what most going back to the risk profile, most company's risk profile is they are not aligned. I challenge you to show me alignment between the SC solution and the average customer. Where is their security threat so high that it justifies changing out this architecture. And now we move to the actual implementation, I said, there's an implementation cost. We've got a workforce that is barely cognizant of basic it fundamentals. Now we are going to be implementing WebSockets, we are going to be implementing TLS, potentially, oh auth, we're going to be implementing authentication certificates. Can you imagine the communication issues and the troubleshooting issues and the configuration issues that are going to occur because of this?
Phil Zito 26:27
Already, people struggle with setting up BBM, DS and bdts. And those are basic architectures. And now we're going to take a technology stack, which by the way, is encrypted, and we are going to go and try to implement it and we're going to run into issues. And then if you have certificate version mismatching, you have issues with potential firewall issues, all sorts of stuff that could cause implementation challenges. Not to mention that a lot of this technology stack, like I mentioned, is all bundled together. So if there's a vulnerability with one of the protocols in that stack, you have to wait for that firmware update to fix that. Hopefully, they've decoupled that, hopefully, they've broken it into microservices, or maybe containerized certain aspects of the protocols so that they can be individually updated. I highly doubt that's happened. Alright, so I mentioned BACnet should go away. And I'm a proponent of that. So let me leave you with a challenge here. And then we'll discuss it a little bit, because I realized we're running up on the 30 minute time limit that I try to keep these episodes to. But if I told you that you had to create a way for building automation systems to communicate, and you could not use a standardized data model, how would you do it? Probably freak out, right? Because every time I say back net, needs to go away, people are like, well, BACnet defined services, you can't use WebSockets. by themselves. You can't use API's by themselves. You have to define services, you have to define data types, but yet you see space after space after space that is using, you know, I don't think restful API's necessarily are the solution. For back net, I just use them as an example oftentimes, because that's something everyone seems to understand. And it's an easy way to say API. But if we constrain ourselves to limitations, like, Hey, we need to use stuff that's only in the space. With it right now. It cannot be a algum ation of multiple different protocols, and has to be something that the code base is open to anyone. So it's not firmware dependent, things like that. What would that look like? I don't profess to have the answer. I am not trying to build a solution to replace back net, I will just tell you that I've seen folks like dystek controls with their API solutions. I've seen a lot of the digital solutions, we're going to be working to get Johnson Controls talking about their open blue solution, which I'm interested in learning more about, because I want to understand how they're doing data, how they're doing API's, how they're handling that. And we're gonna be also trying to get increasingly more people who are outside of our industry, but are kind of in our industry, and how are they doing data? How are they doing their application stacks, I want to start digging into this. Because we often say that we want open systems, we often say that we want to go and have open communications and not be tied down to someone. But yet we're following the same path we've done time and time again. And the reason I'm hammering this so hard lately is because I really feel like we're about to see a surge of retrofits, you know, we're talking with marine ehrenburg. And I think that's Episode 240. And she was with we were work she was with jL she is with a bunch of different companies think Seabury as well, she has a very interesting look at commercial assets, and just buildings in general. And one of the things she said is that we're going to see people consolidate and then modernize their existing assets that are high performers. In order to modernize that often requires new control solutions. And if we're rolling out yet another protocol, that is going to make it difficult for developers, that's going to make it difficult for data consumption and sharing. And that is going to constrain us to a data model that is not really an IT standard data model, then I think we're going to be in for a world of hurt, my hope is to get the folks from haystack and brick schema on the podcast as well. So we could talk about their data stacks. I see, I feel like that is where we should be going. As an industry, we should have a transportation method. Right. And that should be a standard it transportation method, we should have standard services. And we should have a data set that is by some outside party that is not going to be wrapped into the network communication piece.
Phil Zito 31:18
I know this was kind of deep in some aspects was kind of high level and others, we bounced around a lot. I really look forward to seeing your dialogue. If you disagree with me, I absolutely encourage you to communicate. I would love to hear your disagreement. I don't take anything personally. And I realized by me saying back that SC is a bad idea that that probably insulted a lot of people. And that was not my intent. I realized there were probably a lot of people who put a lot of hours into creating the solution. They're like, who the EFF is this, Phil Zito guy telling us our thing is not good. And I hope that I was fair to the solution at the beginning of this episode by saying that it's not a bad solution. Technically, it's sound, it does a lot of good things. I feel based on analyzing the market, that it's a solution designed for a very specific set of customers. And it's not really applicable to the security needs of the greater market. And it's going to provide cost of material or labor and material costs, it's going to actually create labor and material costs that are not necessary and become a burden to both those on the construction as well as on the service side. So my hope is that you'll consider that. You'll think through this. You'll have your thoughtful comments and let's dialogue about it. I'm always open to admit where I'm wrong. Or maybe somewhere I've missed. I don't know everything. But I will tell you looking at this with no dog in the fight, it does not look like a good solution for the mass market. So that being said, I encourage you to go to podcast smart buildings Academy, Ford slash 238. Once again, that is podcast dot smart buildings Academy, Ford slash 238. I'd love to see your comments. I'd love to have a conversation with you. Thanks a ton. Have a great rest of your day.
Transcribed by https://otter.ai
