Open Source Network Security with Cumulus Linux

Networks are changing. More and more we’re hearing terms like whitebox, britebox, disaggregation, NOS, commodity hardware, and open source when we talk about the future of networking. I’ll assume you get that and spare you a description of these terms here. If you do want a crash course on network disaggregation and how it relates to orchestration/SDN, check out my previous post on the Packet Pushers blog.

With that bit of housekeeping out of the way, let’s dig right into today’s topic: Open source network security.

First, why does security matter? If you’re like most network engineers, your primary goal typically is to get bits of data from one place to another. Anything that interferes with the free flow of packets and frames is a potential problem. So the goals of security can at first appear contrary to those of the network. Raise your hand if you’ve ever been frustrated by a firewall rule or some seemingly arcane security policy!

Unfortunately, we no longer have the luxury of ignoring security. Today’s network is one of the most crucial pieces of IT infrastructure for any organization and for the economies we operate in. It simply must work, which leaves us no time to deal with interruptions from malicious activities or bad actors.

In fact, if we look at security from the highest levels, it’s all about three things: confidentiality, integrity and availability. Even the most hard-nosed network engineer should recognize these ideals. Isn’t that exactly what is needed to design, build and operate a trusted, reliable network?

This probably isn’t news to you. The demands for security are mounting just as quickly and forcefully as the demands for a more pervasive, resilient network. So how do you meet these demands? One way to build a flexible, cost-effective and programmable network is to leverage disaggregation and open source software. But is that safe? Can we have security and flexibility? Can we build an open source network for mission critical applications, or do we need to rely on manufacturers who provide proprietary software to ensure the level of security needed in today’s networks?

In this post, I’ll use Cumulus Linux as a case study for secure open networking, and answer those exact questions.

Open Source Security

Before we can answer the question of open source software’s impact on the security of a network, we need to look at the security of open source itself.

As you know, the defining characteristic of open source software is that the source code is made publicly available to all. This is in contrast to proprietary, closed source software where the company who writes the code keeps it a secret.

This distinction is also the root of the most contentious security debate between open source and proprietary software in general.

Proponents of proprietary software will tell us that publishing the source code of open source software allows bad actors to easily identify, understand and exploit security flaws in the software. Anyone can download and examine it! They will go on to say that proprietary software is thus safer, because the code is a secret, and that it is much harder to reverse engineer security flaws than it is to simply find them in open code repositories. OK, makes sense.

On the other side of this debate are the open source proponents who will tell you that sunlight is the best disinfectant. That is, that more eyes on the code means better software because a wide and diverse community of developers is able to review the code, identify flaws and fix them directly. No need to wait for the vendor to accept your request and make the needed changes. Now that’s interesting!

This open versus closed debate is one that seems to have been largely settled in the cryptography space and captured in what is called ‘Kerckhoff’s Principle,’ which reads; “A cryptosystem should be secure even if everything about the system, except the key, is public knowledge.” Meaning that the best system does not rely on secrets to be secure, but rather on robust engineering. This is essentially the opposite of ‘security through obscurity’ and was further refined by Claude Shannon into the following maxim: “one ought to design systems under the assumption that the enemy will immediately gain full familiarity with them.“

Bruce Schneier, a world renowned security expert, explained this in a newsletter, back in 1999:  “In the cryptography world, we consider open source necessary for good security; we have for decades. Public security is always more secure than proprietary security. It’s true for cryptographic algorithms, security protocols, and security source code. For us, open source isn’t just a business model; it’s smart engineering practice.” He goes on to push back on the idea that being public is itself a flaw: “Public algorithms are designed to be secure even though they are public; that’s how they’re made. So there’s no risk in making them public. If an algorithm is only secure if it remains secret, then it will only be secure until someone reverse-engineers and publishes the algorithms.” Finally he talks about the speed with which flaws can be worked out of open versus closed source systems: “So a two year-old piece of open source code is likely to have far fewer security flaws than proprietary code, simply because so many of them have been found and fixed over that time. Security flaws will also be discovered in proprietary code, but at a much slower rate.”

There is, however, a potential flaw in this argument when we’re talking about open source code in general. In order for openness to result in additional security (and fewer bugs overall), we need to ensure that it is in fact being reviewed. Luckily for our case study, Cumulus Linux is based on Linux, one of the most reviewed open source code bases on the planet. This is true of most open NOS’s, so it’s worth taking a look more specifically at Linux itself.

Linux Security

In addition to being an open source operating system, Linux is one of the most prominent and popular examples of free and open source software ever.

Knowing that, it is likely not surprising that the Linux model creates security through transparency. With its large base of developers from all over the world, across many industries, communities and organizations, we can be fairly certain that Linux is getting all of the expert attention needed to ensure swift identification and eradication of security flaws.

Debian Linux

One specific detail that is pertinent to our current case study is that Cumulus Linux leverages the Debian distribution of Linux. It is instructive then to take a look at the Debian Linux security policies: “Debian takes security very seriously. We handle all security problems brought to our attention and ensure that they are corrected within a reasonable timeframe. Many advisories are coordinated with other free software vendors and are published the same day a vulnerability is made public and we also have a Security Audit team that reviews the archive looking for new or unfixed security bugs. Experience has shown that security through obscurity does not work. Public disclosure allows for more rapid and better solutions to security problems.”

With a wide community developing, testing and supporting Debian, it becomes nearly impossible for bad actors (think state actors, malicious hackers and others) to insert backdoors or other intentional security flaws because they would simply be caught by the experts who are constantly reviewing the code. Contrast this with proprietary systems, where only a privileged set of employees from one group in one company has access to the code. It appears obvious which system has the potential to be more secure.

Linux Security Tools

In addition to all the expert eyes pursuing the code, reviewing changes and providing transparent security announcements, Linux offers many native security tools. These tools are available to any system based on a full and open Linux distribution. Let’s look briefly at a few of them.

A pluggable authentication module (PAM) is a mechanism to integrate multiple low-level authentication schemes into a high-level application programming interface (API). It’s a one stop shop for authentication that has been available as a stand-alone offering since 1996. Imagine how many expert eyes have reviewed and ensured its robustness in those twenty plus years! Cumulus uses PAM to control the length and strength of passwords and ensure solid defaults. This can be enhanced with packages such as the libpam_pwquality and pam_cracklib, which are available on the Debian repository.

Another password related tool is the libsnmp-dev package that can be installed on an open Linux device to prevent the use of cleartext passwords with SNMPv3.

While strong passwords can help keep unauthorized users from logging into a network device, we also have to be vigilant of many other forms of attack and malicious traffic. To view and control the ports open on a Linux device, we have tools such as netstat and systemctl. There is also a suite of Linux netfilter tools (iptables, ip6tables, and ebtables) to identify and filter specific IPv4, IPv6 and layer 2 datagrams. Oh, and don’t forget the denyhosts utility (available on the Debian repository) for helping to prevent SSH attacks.

Beyond these, there is a list of additional tools too long to dig into here. Things like snort, pfsense and clamav are all available to increase the security of an open network device and the networks it is connected to.

Proprietary Linux

Some of you may be thinking, “So what? My favorite big name IT vendor uses a Linux/BSD kernel, so I’m already getting all of this, right?”

Well, not necessarily. Just because something uses a *nix kernel under the hood does not always guarantee the same advantages that an open NOS provides. In fact, it is quite likely that the very act of making the operating system proprietary actually negates many of these advantages.

Right off the bat, you lose the ability to immediately apply community patches to discovered security flaws. Because the system is now proprietary, you are forced to wait for the vendor to identify, accept and test the patch on their modified version of the software. Think of the lag between pure vanilla Android updates and those provided by large phone manufacturers that repackage Android into their own particular flavor.

Even worse, the folks who implemented the proprietary system could have included old versions of open source packages, or made other mistakes that make the software less secure. And because these proprietary systems are closed source, there is no easy way to examine the code to identify such missteps. We have to just wait for someone to reverse engineer the software, or otherwise exploit the flaw. That is obviously less than ideal. And what about intentional weaknesses? Can you really trust any code that you aren’t allowed to examine for yourself?

Another major factor to consider with proprietary software, whether based on Linux or not, is your access. With an open NOS, you have full access to see and change anything you need. The level of visibility and granularity of control is hard to beat in any proprietary system. An open NOS also gives you the ability to install and use the widest array of additional packages and tools. Have you ever tried to run snort on a legacy monolithic networking device? Chances are you couldn’t even attempt it.

There are two caveats we need to address here:

1. Networking vendors are obviously not offering closed source proprietary software with the intention of making it less secure. So why are they? One reason to consider is that by closing the system, offering less visibility and control, they are actually making it less prone to user error. Call this the guardrail effect, and be sure you are ready to own your destiny if you dive into fully open systems.
2. Many traditionally closed operating systems are being opened up to allow more access to run third party tools, use APIs, etc. Rather than a knock against open systems, this seems to be a validation of the model, and an indicator of where the future might lie.

Whether a vendor builds proprietary software for more control, more profit or more features, the fact remains that closed source systems are islands that will never be as fully explored as open source systems. There is just no way that more people are using any one proprietary NOS than are using Linux. Thus, there will always be more expert eyes looking out for (and correcting) problems in the latter than in the former.

Cumulus Linux Security

So far we’ve seen that open source software can be just as secure as proprietary software as long as there is an active community of experts reviewing the code — and that Linux has such a community. But how does that translate into networking and the security of a network built on open software? I’m glad you asked. Using Cumulus Linux as our case study, we’re going to dive down that rabbit hole right now.

The first thing worth pointing out is what doesn’t change when going from a proprietary NOS to a more open one. Overall, the same security protocols and techniques are used in both. Cumulus Linux uses the industry standard OpenSSH suite, including support for local passwords, LDAP, TACACS+ and private keys. Cumulus makes available, and recommends using SNMPv3 for secure authentication and encryption of management traffic. Need a management VRF to keep that traffic separate? Check mark. And what about routing protocol security? Again, industry standard implementations of BGP, OSPF, BFDS, MD5, TTLs, etc.

Perhaps a bit more interesting and unique is how well Cumulus Linux is hardened by default:

• ZTP is disabled by default after the initial configuration process is complete. If you are reading this in early 2018, you’ll know exactly how important that is!
• Telnet, ftp, tftp and other less secure access methods are disabled by default.
• Cumulus Linux 3.6 and newer ships with direct login to the root account disabled by default.
• If local passwords are used, the obscure option is enabled by default, which requires a password of at least 6 characters in length, requires complex passwords and does not allow re-use or similarity of the old password for all non-root user accounts.
• Cumulus provides control plane protection by default, and offers an additional method (The default filters are written directly into the hardware on the switchport interfaces, whether the protocol is active or not, and perform at line rate; no performance degradation occurs).
• All logging in Cumulus Linux is done via the rsyslog mechanism with high precision timestamp logging on by default (Logins and attempted logins are logged by default in /var/log/syslog).
• Cumulus does not enable SNMP by default for security reasons (To increase security by default, Cumulus Linux versions 3.6 and later only listen to SNMP traffic coming from the switch itself — accessing SNMP information from outside the box is disabled).

Beyond secure defaults, Cumulus Linux offers several options for ensuring additional security. Here are a few that jumped out at me:

• Secure Package Management: All Cumulus Linux packages are GNU Privacy Guard (GPG) signed. Also, you can Apt-get to a Cumulus repository using https for secure transfer of Cumulus packages.
• Enhanced Netfilter Abilities: Cumulus Linux goes further than Debian Linux by supporting additional targets such as SPAN, police and tricolor police.
• Prescriptive Topology Manager (PTM): This is a dynamic cabling verification tool that uses LLDP and an uploaded diagram to help detect and eliminate layer 0 errors. The check is performed on every link transition on each node in the network. It can also be used for OSPF unnumbered verification.
• Hardware-Based DDOS Protection: This is only applicable to switches with a Broadcom Trident, Trident II or Tomahawk ASIC, so it is disabled by default.

Summary

I can hear you from here: “That’s great, Chris, but what does it all mean?!”

Before I answer that, I want to take a moment to talk a bit more about security in general.

When it comes to security, whether it’s network security, cloud security, endpoint security, open source security, closed source security, or even personal security, there are no silver bullets. There is no single answer, no magic pill, no miraculous product that eliminates your need to plan thoughtfully and observe diligently. The most secure software on the planet is worthless if you fail to patch vulnerabilities as they are discovered. And they will be discovered. Likewise, all the security products and services on the planet are worthless if you don’t properly configure and implement them.

All that is to say that ultimately, security is your responsibility. That is true when you buy proprietary software and when you implement open source software. It’s true when you install physical devices and when you consume cloud resources. It is simply the way of the world.

So, is open source software a network security risk? Only if you let it be. Linux, Cumulus Linux, and many open source projects provide you with all of the tools you need to operate a secure network, but at the end of the day, it’s up to you to ensure they are used properly.

If you want to learn all about how to secure Cumulus Linux for yourself, check out this whitepaper, and if you want to stay up to date on all things Cumulus security, subscribe to this mailing list.

This post first appeared on the Cumulus Networks blog.