Secure the "Internet of Things" (IoT)

I'm going to digress from my usual posts about security and the rambling of notes for my personal projects into an actual article about something of substance: the Internet of Things. To those of us connected to the Information Security field we know how insecure these systems can be and is often the bane of our existence. I, along with many others, keep saying how bad IoT vendors are about securing their devices, to the point of wanting legislation, so I'm going to briefly play devil's advocate and put myself in their shoes. I find this process is helpful for my own understanding of how something works and hopefully, if you're an IoT developer, get an idea of our perspective.

So, the background first... I'm in the midst of developing a product & service that relies heavily on the Raspberry Pi platform and specifically the ODROID-C2 incarnation of the RPi. My requirements lean heavily on open source security tools and rather than making all of them work on Raspbian I chose to go with a security based Linux, namely Kali. Kali runs on the ARM platform, both the one that comes with the RPi-3B+ as well as the chip that comes on the ODROID-C2. Add to that the scripts to build the ARM version are open source. So, I decided to take the base Kali ARM system, modify it to suit my needs, and build my own version of it from scratch. This is very much an IoT project and is similar to what many other people need to do in order to get their software to run on the hardware out there. So, now I'm going into 'why' I have found the IoT world is so insecure and maybe a little on what we can do about it.

This brings us to the first thing I thought about: nearly all of it is homegrown software. If you are going down the Linux path you will be building things on your own. I'm also guessing many of the IoT developers don't have what I do, namely 20+ years using and managing Linux systems. As a result of needing to build from open source software, usually on your own, there's a ton of room to make mistakes. I get that, really, I do. But at the same time if you're one of these developers you should also realize this. Admit you're probably out of your depth and get help. Yes, it might cost real money and yes it might add time to your project but by all means do it. I myself was quite rusty on the internals of Debian (which Kali is based on) so I took a month to tear apart the basic process of building a Debian install (debootstrap, etc).

The next thing I noticed was a distinct lack of updated software, this is even more apparent in the IoT hardware world. I chose the ODROID-C2 because it comes with 2GB of RAM whereas the RPi-3B+ comes with 1GB and I need the additional RAM. This choice means I have to settle for whatever Linux distributions I can find that will work on this hardware, in my case Kali was one of the options. I will use a specific example of outdated software. When I started digging in to Kali ARM and the installer scripts I found that it uses an older kernel, 3.14, which puts it at 3-4 years old. A quick look at the Linux kernel site shows there were something like 79 revisions in the 3.14 tree, I'm guessing at least one of those was a security update/patch. So, why not just use a newer kernel you say? Glad you asked. I experimented with going to the latest 3.x release, 3.16.56, and it failed to compile. And that's just a small incremental update, i'm sure going to the 4.x kernel could be major surgery. Looking around the Kali ARM Github area I can see that issues go for months without responses, that there are only a handful of people working on it and of those only a couple appear to be active. I'm guessing support will be, ah, spotty at best.

So, what do you do? Continue to use a version of software that likely has vulnerabilities? Switch to a different distribution that might be more supported? Commit the time to make the latest release work (and donate it back)? I'm not sure which option I'll take at this point but it will likely be patching the more recent version and making it work. But what happens to other people using this, or comparable systems, that don't know how to do that? My guess is they just go with what works, possibly oblivious to the fact that what they're going to release is likely insecure. About that refrigerator running a 7 year old kernel...

As I have been developing this product I keep coming into situations where I am staring at what I consider to be "best practices" for security and catch myself thinking (and saying) "I'll work it out when I harden the device" which leads me to... You DO have a plan and time set aside for system hardening and security testing right? That's the phase in which you pretend like you're a bad guy and try to cause harm to your baby, seeing how much you can break in the process. Which leads me to... You DO have the ability to do this right? If not, again, GET HELP if you don't. There are people and companies out there that can do this. Yes, it will cost money and yes, it will take time but really, just do it. I have a suspicion many of the insecure devices we see never went through this phase. I'm also willing to bet many of the people that developed them didn't even know this was even a thing.

Now, the problem with the "I'll deal with it during hardening" is that things get missed, it happens, we're human and make mistakes. To counteract this you MUST fight the urge to "punt" until later and adopt secure practices throughout the entire development lifecycle. When gathering requirements, which I'm hoping you actually do, bake security in there. When you're actively developing use secure practices in your code, tools and processes. You DO know how to do this right? By now I hope you're getting the point here: get help when you need it. Again, yes, it might cost money and yes it might add time but you know it's the right thing to do.

Note how all the above represents a process of continuously asking the question "is what I am doing secure" and addressing those issues when they come up. I'm not punting to the next version to fix something, it gets done now.

/rant

Installing RabbitMQ on CENTOS 7

Good guide, works out of the box correctly.

https://www.vultr.com/docs/how-to-install-rabbitmq-on-centos-7

ImportError: No module named MySQLdb

pip install mysql-python

error installing libmysqlclient-dev on kali linux

When trying to install the libmysqlclient-dev package on Kali you get the following error:

Package libmysqlclient-dev is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source

E: Package 'libmysqlclient-dev' has no installation candidate

Install apt-get install default-libmysqlclient-dev instead

Python Celery MySQL results error

While trying to store Celery results in a MySQL database I received the following error:

Native table 'performance_schema'.'session_variables' has the wrong structure")

After some searching I found something that said to run 'mysql_upgrade -u root -p' then restart MySQL, this fixed the issue for me. The order in which things were done:

• MySQL was installed using brew on OS/X 10.13.4 (mysqld  Ver 5.7.22 for osx10.13 on x86_64 (Homebrew))
• mysql.server start
• mysql_upgrade -u root -p
• mysql.server stop
• mysql.server start

The Celery code:

#!/usr/bin/env python
from celery import Celery

app = Celery('tasks', \
        broker='pyamqp://guest@localhost//', \
        backend='db+mysql://root:password@localhost/pigpen')

@app.task
def add(x, y):
    return x + y

Celery output worker side:

celery@dy-mac.local v4.2.0rc3 (windowlicker)

Darwin-17.5.0-x86_64-i386-64bit 2018-05-20 15:20:01

[config]

.> app:         tasks:0x10e073590

.> transport:   amqp://guest:**@localhost:5672//

.> results:     mysql://root:**@localhost/pigpen

.> concurrency: 8 (prefork)

.> task events: OFF (enable -E to monitor tasks in this worker)

[queues]

.> celery           exchange=celery(direct) key=celery

[tasks]

  . tasks.add

[2018-05-20 15:20:02,927: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//

[2018-05-20 15:20:02,941: INFO/MainProcess] mingle: searching for neighbors

[2018-05-20 15:20:04,071: INFO/MainProcess] mingle: all alone

[2018-05-20 15:20:04,117: INFO/MainProcess] celery@dy-mac.local ready.

[2018-05-20 15:20:32,312: INFO/MainProcess] Received task: tasks.add[9a6c0983-64cf-4053-a64f-04894704badf]

[2018-05-20 15:20:33,263: WARNING/ForkPoolWorker-5] /Users/dyoung2/anaconda/lib/python2.7/site-packages/sqlalchemy/engine/default.py:507: Warning: Invalid utf8 character string: '80024B'

  cursor.execute(statement, parameters)

[2018-05-20 15:20:33,266: INFO/ForkPoolWorker-5] Task tasks.add[9a6c0983-64cf-4053-a64f-04894704badf] succeeded in 0.950585585s: 6

Python client side results:

Python 2.7.13 |Anaconda 4.3.1 (x86_64)| (default, Dec 20 2016, 23:05:08)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> from tasks import *
>>> result = add.delay(3,3)
>>> result.status
'SUCCESS'
>>> result.ready()
True
>>> result.get()
6

Python notes

A few interesting Python things that I always forget how to do. The problem is I don't write Python code on a daily basis, nor have I done so for many years like I did Perl, so I tend to forget things in between the times I do write it. I'll add more here as I come across them.

Converting int to a string inside a print statement, note the %(var)s part.

print("\nServices running on localhost:%(port)s\n" % {'port': port})

Catching CTRL-C:

def signal_handler(signal, frame):
        print('You pressed Ctrl+C!')
        print("Exiting...")
        sys.exit(0)

signal.signal(signal.SIGINT, signal_handler)

Full absolute path for a log file

logfile = join(dirname(normpath(abspath(__file__))), 'file.log')

OpenVAS Performance Tuning

As part of a project I'm working on right now I wanted to know what the "optimal" settings were for running OpenVAS on an ODROID-C2 Raspberry Pi type system. This single board computer (SBC) has 4 CPU cores and 2GB of RAM along with a 1GB ethernet adapter. My test system is running Kali Linux, 2018.2, with all current updates before being tested.

For the tests I picked one scan target to run the jobs against, a single IP address on my network that I know had a few minor vulnerabilities. I wanted to test the various settings such as max_hosts, max_checks, be_nice and whether or not the host was already in the database. I made one change at a time and submitted the job to the default scanner using the "Full and fast" scan type each time. My submission XML looks like this:

<create_task>
  <name>New 10.0.0.9</name>
  <comment>cmdline scan</comment>
  <config id="daba56c8-73ec-11df-a475-002264764cea"/>
  <target id="5b9c2dde-77ab-4511-b439-c96984843f5b"/>
  <preferences>
    <preference>
        <scanner_name>max_checks</scanner_name>
        <value>12</value>
    </preference>
  </preferences>
</create_task>

My script to submit that to the scanner is:

#!/bin/bash
#
RETVAL=$(cat create_task.xml | omp --xml -)

TMPTASK=$(echo $RETVAL | awk '{print $2}' | awk -F= '{print $2}')
TASK=$(echo $TMPTASK | tr -d \")

RETVAL=$(omp --start-task $TASK)

I tracked the results of each scan, here's the raw data:

max_hosts max_checks be_nice num hosts host in db? alterable run time min 4 2 yes 1 yes yes 31 4 4 yes 1 no yes 26 4 4 yes 1 yes yes 26 4 4 yes 1 yes yes 25 4 4 no 1 yes yes 25 4 6 no 1 no yes 24 4 6 no 1 yes yes 25 4 8 no 1 yes yes 24 4 10 no 1 yes yes 30 4 12 no 1 yes yes 50

And a couple charts based on that data:

What did I find?

• Whether or not the host is in the database doesn't appear to impact performance. There appears to be no appreciable benefit on a single host scan if there is existing data.
• The be_nice setting in openvassd.conf doesn't appear to have an appreciable impact on performance.
• After 6-8 checks per host there is no noticeable positive effect on performance, it plateaus around there then starts to drop, by 12 performance is BAD. But remember, this system only has 4 cores in it.
• Load average on the system went pretty high during the scans, as much as 5 at some points but the system was still useable while at that level. I'm amazed at these little boxes. vi would hesitate a bit on load but nothing terrible.

So, what happens if you want to scan a dozen hosts? If you're using only one ODROID I'm recommending you don't exceed 8 active plugins running at any point in time, so you'd set max_hosts to 2 and max_checks to 4 or just allow 1 host at a time and 8 checks, up to you. I plan on testing multiple hosts next, we shall see if there's any benefit in running more than one host at a time. The other option? Split that scan up across multiple ODROIDS using something like Celery and RabbitMQ, that's what my service is built on.

Caveats, notes and todos

• The target I tested had very few minor vulnerabilities though OpenVAS runs whatever was specified in the scan config regardless.
• Remember, this system only has 4 cores.
• Need to run the same tests against a VM that has vulnerabilities, Metasploitable or DVWA.
• Test multiple hosts at the same time to see where things get crappy.
• No idea if the decreases in performance around 10 are due to I/O, I wasn't monitoring it but my gut says no, it's not.

Metasploitable exploitability guide

A good intro to Metasploitable: https://metasploit.help.rapid7.com/docs/metasploitable-2-exploitability-guide

Metasploitable root access

To gain root access to metasploitable you first need to log in to the console as the user msfadmin with the same text as the password (msfadmin). From there do an sudo su, enter that same msfadmin password and away you go.

OpenVAS Commands

This is about the best reference source I've found for reference on the XML you need to create for submitting commands via the omp command.

http://www.openvas.org/omp-2-0.html

Change OpenVAS password

For some reason my last automated install of OpenVAS didn't catch the creation of the random password at the beginning, here's how you can change it manually:

openvasmd --user=admin --new-password=<new_password>

OpenVAS on all interfaces

By default OpenVAS only listens for connections on 127.0.0.1, if you want to access it off-machine you'll need to do this:

cd /lib/systemd/system && sed -e 's/127.0.0.1/0.0.0.0/g' greenbone-security-assistant.service openvas-manager.service openvas-scanner.service -i.BAK

After that do a systemctl daemon-reload and either reboot or restart those services using systemctl. The grep command will create a file of the same name with a .BAK extension in that directory should something go wrong.

OpenVAS port 80

By default the OPenVAS security assistant listens on port 80 and redirects connections to port 9392, this causes issues if you want to run a web server on the same machine. This manifests itself with the following showing up with lsof:

root@pi-5785:/var/log/openvas# lsof -iTCP -sTCP:LISTEN
COMMAND     PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd        349     root    3u  IPv4  12027      0t0  TCP *:ssh (LISTEN)
sshd        349     root    4u  IPv6  12029      0t0  TCP *:ssh (LISTEN)
gsad      12465     root    5u  IPv4 471034      0t0  TCP *:9392 (LISTEN)
gsad      12468     root    5u  IPv4 471736      0t0  TCP *:http (LISTEN)
openvasmd 14004     root    4u  IPv4 479681      0t0  TCP *:9390 (LISTEN)
postgres  16285 postgres    3u  IPv6  36152      0t0  TCP localhost:postgresql (LISTEN)
postgres  16285 postgres    6u  IPv4  36153      0t0  TCP pi-5785:postgresql (LISTEN)

Note gsad listening on port 80 there as well as 9392.

To completely disable port 80 on Kali Linux, and only use 9392, edit the file /lib/systemd/system/greenbone-security-assistant.service and add the --no-redirect option to the ExecStart part:

ExecStart=/usr/sbin/gsad --foreground --no-redirect --listen=0.0.0.0 --port=9392 --mlisten=0.0.0.0 --mport=9390

After that run systemctl daemon-reload then systemctl restart greenbone-security-assistant and port 80 won't be listening.