Craig DiLouie, LC, CLCP recently had the opportunity to interview Jonathan Cartrette, Systems Architect, Wattstopper/Legrand for an article that will be published in 2019 for ELECTRICAL CONTRACTOR. The topic? Lighting and cybersecurity.
DiLouie: How big an issue is security for networked lighting control systems?
Cartrette: Big, and bigger with wireless.
The question, “Why is security important for a light-switch, and is it really worth all the effort?” is not about ransomware locking out your light switch or a disgruntled ex-employee flicking ON and OFF the lights. The concern that control system can be a jumping off point for malware and attacks that do matter because they leak sensitive, exploitable data over years. One of the more notable recent examples was a thermometer in a fish tank in the lobby of a casino. No one cared about the fish and, thankfully, no fish were harmed in the making of this hack, but the network this IoT thermometer used was connected closely enough to the enterprise network in the casino that this initial exploit was critical towards the execution of the final attack – which did real financial damage!
DiLouie: How strong is this an inhibitor to adoption?
Cartrette: Wireless security is not well known to key customers, which keeps them away. And smaller customers rarely appreciate the situation enough to care.
There’s a lose/lose situation in the market regarding security of wireless networked lighting controls, which is based on the size and technical acumen of the customer.
For large, technically savvy customers, the worry is that wireless networked systems implemented by lighting companies and building controls manufacturers – the whole lot of us! – must be inherently insecure because there have been so many glaring revelations about “Security 101” errors. The video a few years back of a hacker’s drone wirelessly controlling lights in a tech building didn’t help. Because of this belief the enterprise customers – the very ones that appreciate the benefits that come from connectivity and access to data – fear wireless systems and are not concerned with the extra cost of wired systems.
Yet on the other hand, security is rarely a significant worry for smaller customers – it’s the “what, someone’s going to hack my light bulb” syndrome. And if they are worried they’ll go with wired systems and just double down on controlling physical access with locked doors, badge access, etc. Ergo, wireless controls would take away my “air gap” to the system; how dumb do I have to be to let my control signal out the door?
DiLouie: What are common concerns?
Cartrette: The ones that are truly common aren’t spurring appropriate action.
I think the answer varies broadly based on the boundaries which define the “commonality” of a concern. This is a big problem if not THE problem.
The “common concerns” tend to sound like an episode of 24 where, to aid in a daring escape the hero/villain hacks the lighting system of a building and plunges everything into darkness (end scene).
Yet for folks who know the “art of the possible,” their main concern is that control systems – through their contact with occupants, service and maintenance personnel, and the manufacturers’ systems – are not the ultimate target of exploitation, they’re an entry point. They’re the first domino in a chain of attacks till the hacker gets to where the valuable data is stored – credit card or social security records, financial info, health care records, hardware schematics, etc.
The Lighting Control industry should accept that we’re a stepping stone to a larger exploit. Once we admit that EVERY system is a possible attack vector to an upstream Enterprise System, we realize we shouldn’t be trying to solve the problem by ourselves. Instead we should borrow from or partner with companies that have created best practices – and those may be from other industries.
DiLouie: How well are these common concerns being addressed by channel partners like electrical contractors?
Cartrette: Not well since the contractors’ role is currently miscast. They’re accepting significant responsibility and risk that few are aware of, and fewer still are properly trained to address.
As the industry increases the coverage for remote expert support to supplement on-site facilities management teams struggling for funding – remote support that literally depends on networked control systems for technicians in a NOC to fix or tune lighting behaviors from distant places, 24/7 365 – then the inherent security those devices and networks offer becomes more important. There are more people to phish, there are more entry points, and there are more interfaces designed for the very purpose of being accessed by our ubiquitous mobile computing devices.
DiLouie: What responsibility does the electrical contractor have for setting up and ensuring security for networked lighting control systems?
Cartrette: The contractor is being forced to accept too much responsibility. Stop doing this.
The contractor is often responsible for setting passwords and making sure the network they set up is locked down once they leave. I’m of the opinion that it’s unconscionable for a control system to punt a topic as complex as cyber security to the tradespeople who were awarded the contract to install hardware.
The home security system monitoring industry and the broadband internet/TV industry are two notable examples of industries that have solved this scope inequity. Both hire and train installers for a very physical, tangible task set. There is technology involved, but the tasks are about mounting window break sensors, terminating control wire and coax, installing power supplies and managing a tight schedule that often gets out of sync. Also imagine they must create a bunch of passwords featuring numbers (but not 3), letters (but not B, R, or E), symbols (but not %, $, or ^), and they need to be 32 characters long, can’t be the same as the last 1000 houses they’ve been to in the past 3 months. It sounds ludicrous, but this is literally what a lot of lighting control systems, and building control systems in general, are requiring of the installers.
With both the home security and broadband cable industry there was a recognition that it was as impractical to find and then hire certified network admins to do installs as it was to continuously train 100% of installers on the requisite depth of cybersecurity minutiae.
The solution was automation. Computer-readable identities called certificates could be deployed onto every piece of the infrastructure like digital genomes to make every single device uniquely identifiable and traceable from the moment it was manufactured. These certificates are mathematically related to each other in a giant family tree and when any two family members meet in the real world they can prove they’re family. Customers don’t ask the installers to get them the pay-per-view fight for free anymore. The installer can’t do it because they don’t have the password, because there are no passwords.
The technology that shut down cable theft has been in continuous use since ~2000 and infrastructure providers in this space have issued billions of certificates. This technology has also proliferated onto the internet – forms of which we see every day whenever a link in our browser starts with, “https://”. Deployment of “Trusted Hardware” outside of broadband began with Trusted Platform Module (TPM) silicon chips in PCs in the mid to late ‘aughts. Now we all use this form of technology daily when we insert our chip cards into point of sale terminals. Those same terminals that got malware on them when someone spear-phished a poor service technician who happened to work on climate control systems at Target now verify the identity of the bank and the card directly using those chips which reduces (or even eliminates) the dependency on the networks and devices in between being secure.
This “Trusted hardware,” strategy is already showing up in IoT including lighting control systems’ cutsheets from multiple vendors. The next challenge will be educating the industry on how it’s accomplished and to select product knowing not all trusted hardware is as trustable as others. Trusted hardware security strategies will be a learning curve just like dimming curves or, more recently, the so called, “Duck Curve,” but I’m confident our sector will rise to the occasion.
DiLouie: What is the risk to themselves and their customers if they do not properly establish security, or otherwise if the system is not properly secure?
Cartrette: Risks to contractors? Liability for a problem you never should have agreed to own; implicitly or otherwise.
Risks to their customers? Customers are likely unaware that their individual systems were not well secured, yet they’ll bare the brunt of any break in. And increasingly it’s a double problem since dealing with the consequences of the break in, and not having done enough to prevent it, are both punishable offenses (depending on the situation).
One barometer I like to pay attention to is the insurance industry. The underwriters of product manufacturers, buildings, and even people are notoriously good theorists of risk. But the difference between them and technologists, who struggle not to come across as hang-wringing alarmists, is that underwriters must measure risk in dollars. The first realization the electrical channel should have is there exists a global community of professionals whose job it is to quantify the risks of a hack – which includes the technical response to it and the PR. Most of the value chain in networked lighting controls does not seem to be aware this is happening to us, so it’s no wonder few succeed at constructing an ROI around prevention.
The industry needs to significantly change acceptable minimums for device and system security, but right now no one incentivized to do anything beyond nominal improvements. So, on the longer timeframe the risk is to all of us in the electrical channel; that we continue to justify our bad reputation for securing products and systems.
DiLouie: What should contractors be looking for in a control system family to confirm proper security is being provided?
Cartrette: Committed hardware partners who can provide easy to use tools for the hard parts of deployment, support over the building lifecycle, and the capability to offer direct, long-term support to the installer, tenant(s), and building owner.
Manufacturers occupy a place in the value chain that lets us look into the technology future of our channel to an extent. If the brand pushed in the spec cycle is going to leave you responsible both for the final security of the system and the consequences for any breach in security, then demand something better.
Then it doesn’t fall to the contractors for best security practices – it absolutely shouldn’t. How can a contractor solve problems that are most often in a manufacturer’s “black box” hardware, or their provisioning requirements?
The best security practices (whether manifest via an App on a phone or online connectivity to a manufacturer’s remote operations center) are widely known and standardized in other industries, so they should be readily and publicly available in ours. Unlike the escalating spec advantage world, we know good security happens to be completely transparent. On some timeframe solutions to this problem will – and should – look exactly the same. That way we can get back to CRI, Kelvin temperatures, expanded data about the space – all the things that give our role in the lighting control market unique value.
DiLouie: What kinds of questions and concerns will IT departments bring to the project, and how should contractors deal with them?
Cartrette: What concerns will departments raise? A: It’s a huge list because IT departments’ job is to say no – and they’re good at it.
As previously stated, “All IoT is inherently insecure” is a commonly encountered assumption. If this sentiment is present, then the conversations to overcome any difficulties will require a deep understanding of any manufacturer’s hardware.
These conversations will be specific and fully buzzword compliant – UDP/TCP ports, traffic patterns, certificates, revocation, firewalls, VLANs, VPNs, QoS, and 802-dot-whatever may all come up in the discussion.
How should contractors deal with this? Involve vendors and bring to bear their expertise in addressing concerns.
The network lighting controls vendor should be the biggest asset here – between them and their local representatives and partners they should educate and eliminate concerns. This includes providing access to control experts, contractor trainings, schematic examples, Technical Bulletins, “factory people” – everything a controls vendor is already doing to help contractors succeed when the conversation is about UL508, NFPA 70, or California Title 24, should also be extended for cybersecurity. Their actions should ensure that contractors can do what contractors do best – install electrical systems in safe, efficient, code compliant, and comfortable commercial buildings.
DiLouie: Are there standardized approaches, or does security vary from manufacturer to manufacturer?
Cartrette: Yes, but there is a lot we could learn from other industries.
There are standardized methods, yet the penetration of them into lighting as best practices is diverse enough to seem random. AES128 can be a useful measure of encryption strength, but AES128 is useless if the key is 0x00001234 for each device in every system and it’s published online, and the default password to the gateway’s user account is called out in the manufacturer’s installation instructions. So, the best practices are increasingly varied and encompass everything from the installer user experience (UX) to the actual technical merit of the forward security post-install. Presently the best practices are not standardized in networked lighting controls today. That’s why looking at other industries and their solutions to similar problems is so crucial to innovation in our space. We need to leapfrog – let’s look at some other ponds.
DiLouie: What are some of the noticeable similarities and differences in approaches from manufacturer to manufacturer?
Cartrette: Staggeringly wide variance even though the cutsheet promises all seem similar.
The biggest difference is transparency and standards vs. lighting companies’ proprietary models.
The problem is how CSI specification language rewards directly comparable text. Literally if vendor A had, “AES128” and so does vendor B, then there is no criteria by which to rule out either. But are they truly the same? If Vendor A is using 0x00001234 all the time, and Vendor B is taking care to protect and randomize keys, then the text of the spec didn’t tell the whole story. This is like valuing a house built with a nail-gun over a house built with a hammer. Tools are important, sure, but the quality of the overall structure (how the tool was used) is more important. The topic of cybersecurity moves much faster than the language of our industry, and we are forced to market methodologies by listing the tools that were used. This leads both to a watering down of similarities, and slower adoption of new technologies that might be harder to explain. If a new best practice requires too much explanation, then it could take a full spec-cycle refresh – or more – to see it adopted. Until those new methodologies can be bullet-pointed (unlikely with security practices) cutsheets and spec language obscure true product differences.
DiLouie: Are there advantages and limitations for adopting certain open protocols for interoperability and communication between control devices?
Cartrette: Absolutely, through usage of standard methods.
With the interoperability that open standards provide the focus on security and the installation/maintenance UX overall can increase. Proprietary technology leaves out the potential that the best and brightest globally advance, modernize, and maintain the core capabilities of a device or system.
But for trust to matter in the real world, and the digital world, it should not be granted lightly. So what’s happening, and somewhat paradoxically, is that the authentication of devices and subsystems one to another is the new horizon for interoperability. Vendor A and B can literally use the same chips, the same network stacks, and the same application layer, but enforce verifiable trust as a requirement to provision the networks and commission the system. Is it interoperable if the technology is the same but an integrator is unable to install, configure to spec, and “get off the job” until Vendor A and B decide how (and if) to trust each other?
However daunting this new horizon may seem, I believe it is a better position for industry to be in – literally equating digital trust relationships with business trust relationships within the channel – and forming ecosystems of like-minded vendors rather than artificially exclusionary boundaries built through proprietary technology.
DiLouie: How will the Internet of Things impact overall cybersecurity and security for networked lighting control systems specifically?
Cartrette: On the long term it will get better and normalize because of standards, but on the short term some of the usage of these technologies is at odds with the needs of the installer.
There’s a potential issue with IoT and connectivity for every individual manufacturer, which is how do you ensure that your products are secure when they’re called on to interface with devices from other manufacturers who potentially aren’t as diligent about security. How do you make certain your chain is secure when there may be other weak links? Until every device carries with it a digital identity, or comes from a single manufacturer, complete security in this situation is difficult.
Yet IoT is the only hope for good-enough, evolve-able, standards-based security in networked lighting control systems because it brings IT, A/V, and occupants into the “Buy-decision” process. The people that value security enough to fund its best-practice implementation in baseband commercial building infrastructure do not currently even get a look at Div. 26 specs for the space they will occupy. The people whose smartphones, PCs, and data are at risk cannot affect the purchase. Lighting systems that track and count people, perform fall/injury detection, manage asset tracking, and more will perform functions that matter to folks outside our sphere. By tying in the final purpose of the built-environment to the currently inaccessible parts of the value chain, customer opinions on how we protect them and their data will matter more as a result.
DiLouie: Is there a way electrical contractors could rate security, such as a checklist or third-party qualification? What industry standards apply, and how effective are they today?
Cartrette: Not yet because compliance determination methodology is as important as the contents of most standards.
There are a host of these standards, some of which are already obsolete, that may come up in the specification process. ISO-27000 family, common criteria EAL+, ICS-CERT, CVE/CVD, UL-2900, PCI-3.0, FIPS 140-2, DIACAP, to name a few. And within many of these there are multiple levels dependent on the verification process employed. But certifications to these standards fall mainly into two categories – device certification or installed system certification – and until they are consistently employed in industry it is difficult to put these into a checklist.
A device (or system of devices) can be certified for its security capabilities, and it means just that – it is capable of being secure. This looks great on packaging, but if the system is installed with “admin/admin” as the default login and password, any deeper capabilities are wasted. Standards bodies are only beginning to fill this gap, and I don’t think there will be a checklist available anytime soon. I would be cautious with any entities promising too much in this regard.
DiLouie: If a contractor asked you to summarize why security is important and what they need to know to avoid risk, what would you tell them?
Cartrette: This WILL affect your business and likely already has.
Security affects all aspects of your business, whether we’re talking about:
• Total Material to install under your scope being reduced
• Labor during installation and system configuration reduced along with the materials
• Cabling, controls enclosures, and other Infrastructure for the networks going to IT and A/V
• Increased time spent interfacing with other trades, and owner’s decision makers
• Insurance Costs due to increased legal risk