chatr

Getting Started With ModSecurity

| Comments

We all know that publishing a website or web based application is the start to a long journey. A lot of care and effort goes into making sure everything is just right. The idea of being “safe” or “secure” is just that, an idea; a pipe-dream if you will. The sooner you realize that you’re always behind, the sooner you can let go and focus on the things that matter. What matters more than anything is understanding what is going on with your site. This means knowing the basics:

  • Who is accessing your site?
  • How are they using it?
  • Are there any errors while they are using it?

There are lots of other things to take into consideration, but we can start here. Like most things, knowledge is the key to being successful in securing the stuff you put on the web. I have talked quite a bit in the past about secure coding practices, and doing what you can to prevent attacks by improving software security. While this is something that we should always strive for, it’s not really enough to provide us with a good understanding of what attacks might be in play at any given moment. There are a number of tools out there to help with this, but I want to talk about one of my favorite free and open source tools, ModSecurity.

For those of you who are already familiar with ModSecurity, you might be thinking that this is obvious. Just put a web application firewall in place. OWASP even says so. While it is true that web application firewalls can help stop would be intruders at the gates, I don’t actually take much stock in their ability to do so. There have been a few hints that using a WAF might even give away a too much information about you and help attackers zero in on you faster. The question is how do we understand what is going on and still protect ourselves from people trying to break down our defenses? For me the answer isn’t really obvious. I am still working on new ideas every day, but since we have to start somewhere, let’s start with setting up ModSecurity and exploring some of its features.

Initial Setup

All of the examples that follow will be using Ubuntu 12.04 LTS Server and Apache. I simply created a new virtual machine for this experiment and worked from there. You can go about this any way you would like, but I found this to be the fastest and easiest setup of all the distro/web server combinations I tried. ModSecurity does run on IIS and nginx, but it requires quite a few more steps to get going. With all of that said, let’s get to work.

I’m assuming a fresh install of Ubuntu server, so the following packages will be the everything needed. If you’re working from an existing system, you might already have these things installed.

1
2
3
$ sudo apt-get install libxml2 libxml2-dev libxml2-utils
$ sudo apt-get install libaprutil1 libaprutil1-dev
$ sudo apt-get install libapache-mod-security apache2

Now we just need to put the configuration into place:

1
$ sudo mv modsecurity.conf-recommended modsecurity.conf

Along with installing ModSecurity we also want to install the core rule set.

1
2
3
4
5
6
7
8
$ wget http://downloads.sourceforge.net/project/mod-security/modsecurity-crs/0-CURRENT/modsecurity-crs_2.2.5.tar.gz
$ tar xvzf modsecurity-crs_2.2.5.tar.gz
$ sudo cp -R modsecurity-crs_2.2.5/* /etc/modsecurity/
$ sudo mv /etc/modsecurity/modsecurity_crs_10_setup.conf.example  /etc/modsecurity/modsecurity_crs_10_setup.conf
$ cd /etc/modsecurity/base_rules
$ for f in * ; do sudo ln -s /etc/modsecurity/base_rules/$f /etc/modsecurity/activated_rules/$f ; done
$ cd /etc/modsecurity/optional_rules
$ for f in * ; do sudo ln -s /etc/modsecurity/optional_rules/$f /etc/modsecurity/activated_rules/$f ; done

After everything is in place, we need to tell ModSecurity to load the OWASP core rules that we just copied in. Open /etc/apache2/mods-available/mod-security.conf and add the following:

1
Include "/etc/modsecurity/activated_rules/*.conf"

Just in case the headers and mod-security modules aren’t enabled for some reason, we can give them a nudge with a2enmod and restart Apache:

1
2
3
$ sudo a2enmod headers
$ sudo a2enmod mod-security
$ sudo service apache2 restart

With everything in place we can give it a try. First, try hitting your server by ip address. You will notice that everything is displayed properly. Looking at /var/log/apache2/error.log presents an entirely different story though. Let’s take a look at one of the log entries:

1
[Sat Jan 05 20:20:55 2013] [error] [client 172.16.39.1] ModSecurity: Warning. Operator LT matched 5 at TX:inbound_anomaly_score. [file "/etc/modsecurity/activated_rules/modsecurity_crs_60_correlation.conf"] [line "33"] [id "981203"] [msg "Inbound Anomaly Score (Total Inbound Score: 2, SQLi=, XSS=): Host header is a numeric IP address"] [hostname "172.16.39.129"] [uri "/favicon.ico"] [unique_id "UOjfh38AAQEAAAkJAboAAAAB"]

This rule is triggered because we hit the server directly by its IP address. Since our web application will most likely be served from a real name, this is a good rule to keep in place. You might be wondering now why all that happened was a log entry. This is because we haven’t yet told ModSecurity to deny requests that trigger alerts. Let’s turn that on now. Open /etc/modsecurity/modsecurity.conf and make the following change:

1
2
- SecRuleEngine DetectionOnly
+ SecRuleEngine On

Then simply restart Apache

1
sudo service apache2 restart

If you reload the page you will get a forbidden message. You are now stopping potentially bad traffic using ModSecurity. In the next installment we will take a look at the rules that are triggered in the default install and learn how to configure the default install to not deny legitimate traffic.

Reboot

| Comments

It’s been quite a while since I have paid real attention to this blog. I have been really busy lately exploring some fun new ideas, and it’s about time I started sharing again. I decided to push things over to Octopress, so it will take just a bit before I get the old posts ported over. If you are looking for an older post, please bear with me as I expect the migration to take the next couple of days.

MySQL/MariaDB Memcmp Vulnerability, Are You Vulnerable?

| Comments

A vulnerability was recently announced that allowed an attacker to successfully authenticate against a database without the correct password. The issue was a call to memcpy that resulted in a value outside of the standard -128..127 range. To view the full announcement you can visit http://seclists.org/oss-sec/2012/q2/493.

Testing your install

You can do a quick test to see if you are affected. Find the location of your mysql install and fire up gdb.

$ gdb bin/mysql

Next, load up libmysqlclient so you can get the symbol reference for the check_scramble function. This is the offender in the vulnerability.

(gdb) file lib/mysql/libmysqlclient.so
Load new symbol table from "/packages/encap/mysql-5.1.56_percona_server/lib/mysql/libmysqlclient.so"? (y or n) y
Reading symbols from /packages/encap/mysql-5.1.56_percona_server/lib/mysql/libmysqlclient.so...done.
(gdb) set logging file /home/user/mysql.out
(gdb) set logging on
Copying output to /home/user/mysql.out.
(gdb) disas check_scramble
...

Now check for the presence of memcmp in the dump

$ cat /home/user/mysql.out | grep memcmp

If nothing is returned, you are good to go. If you dig a bit further, you will notice that the call to memcmp has been replaced with

repz cmpsb %es:(%rdi),%ds:(%rsi)

which will prevent for this vulnerability from occurring. If you do see the call to memcmp, then further analysis is required. Your specific build of gcc may or may not expose this issue.

Detailed Analysis

You can do a simple test on your system with a short C program

test.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
#include <string.h>
#include <stdio.h>

int main() {
  char *s1 = "b!r";
  char *s2 = "bÿr";
  int result;

  result = memcmp(s1, s2, sizeof(s1));
  if ((result < -128) || (result > 127)) {
    fprintf(stderr, "unexpected memcmp(): %d\n", result);
    return 1;
  }

  return 0;
}

Compile and test:

$ gcc test.c -o test
$ ./test
unexpected memcmp(): -162

Using gdb to dump this, you can see the call to memcmp:

$ gdb test
(gdb) disas main
0x0000000000400534 <+0>:    push   %rbp
0x0000000000400535 <+1>:    mov    %rsp,%rbp
0x0000000000400538 <+4>:    sub    $0x20,%rsp
0x000000000040053c <+8>:    movq   $0x40068c,-0x8(%rbp)
0x0000000000400544 <+16>:   movq   $0x400690,-0x10(%rbp)
0x000000000040054c <+24>:   mov    -0x10(%rbp),%rcx
0x0000000000400550 <+28>:   mov    -0x8(%rbp),%rax
0x0000000000400554 <+32>:   mov    $0x8,%edx
0x0000000000400559 <+37>:   mov    %rcx,%rsi
0x000000000040055c <+40>:   mov    %rax,%rdi
0x000000000040055f <+43>:   callq  0x400430 <memcmp@plt>
0x0000000000400564 <+48>:   mov    %eax,-0x14(%rbp)
0x0000000000400567 <+51>:   cmpl   $0xffffff80,-0x14(%rbp)
0x000000000040056b <+55>:   jl     0x400573 <main+63>
0x000000000040056d <+57>:   cmpl   $0x7f,-0x14(%rbp)
0x0000000000400571 <+61>:   jle    0x400599 <main+101>
0x0000000000400573 <+63>:   mov    $0x400695,%ecx
0x0000000000400578 <+68>:   mov    0x2003d9(%rip),%rax        # 0x600958 <stderr@@GLIBC_2.2.5>
0x000000000040057f <+75>:   mov    -0x14(%rbp),%edx
0x0000000000400582 <+78>:   mov    %rcx,%rsi
0x0000000000400585 <+81>:   mov    %rax,%rdi
0x0000000000400588 <+84>:   mov    $0x0,%eax
0x000000000040058d <+89>:   callq  0x400440 <fprintf@plt>
0x0000000000400592 <+94>:   mov    $0x1,%eax
0x0000000000400597 <+99>:   jmp    0x40059e <main+106>
0x0000000000400599 <+101>:  mov    $0x0,%eax
0x000000000040059e <+106>:  leaveq
0x000000000040059f <+107>:  retq

If you recompile using optimizations, you will notice a slight difference:

$ gcc -O3 test.c -o test
$ ./test
<nothing else happens>
$ gdb test
(gdb) disas main
Dump of assembler code for function main:
0x00000000004004f0 <+0>:    mov    $0x8,%ecx
0x00000000004004f5 <+5>:    sub    $0x8,%rsp
0x00000000004004f9 <+9>:    mov    $0x40062c,%esi
0x00000000004004fe <+14>:   mov    $0x400630,%edi
0x0000000000400503 <+19>:   repz   cmpsb %es:(%rdi),%ds:(%rsi)
0x0000000000400505 <+21>:   seta   %dl
0x0000000000400508 <+24>:   setb   %al
0x000000000040050b <+27>:   sub    %al,%dl
0x000000000040050d <+29>:   xor    %eax,%eax
0x000000000040050f <+31>:   movsbl %dl,%edx
0x0000000000400512 <+34>:   lea    0x80(%rdx),%ecx
x0000000000400518 <+40>:    cmp    $0xff,%ecx
0x000000000040051e <+46>:   jbe    0x400536 <main+70>
0x0000000000400520 <+48>:   mov    0x2003c1(%rip),%rdi        # 0x6008e8 <stderr@@GLIBC_2.2.5>
0x0000000000400527 <+55>:   mov    $0x400635,%esi
0x000000000040052c <+60>:   callq  0x4003f0 <fprintf@plt>
0x0000000000400531 <+65>:   mov    $0x1,%eax
0x0000000000400536 <+70>:   add    $0x8,%rsp
0x000000000040053a <+74>:   retq

Note that the call to memcmp has been replaced by the repeat while zero string operation cmpsb. This will change the result and should actually thwart any bad mojo that was present previously.

If you look at the code in the MySQL project in sql/password.c, the memcmp only happens in one place, the check_scramble function, which has the following code:

password.c
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
my_bool
check_scramble(const char *scramble_arg,
               const char *message,
               const uint8 *hash_stage2)
{
  SHA1_CONTEXT sha1_context;
  uint8 buf[SHA1_HASH_SIZE];
  uint8 hash_stage2_reassured[SHA1_HASH_SIZE];

  mysql_sha1_reset(&sha1_context);
  /* create key to encrypt scramble */
  mysql_sha1_input(&sha1_context, (const uint8 *) message, SCRAMBLE_LENGTH);
  mysql_sha1_input(&sha1_context, hash_stage2, SHA1_HASH_SIZE);
  mysql_sha1_result(&sha1_context, buf);
  /* encrypt scramble */
    my_crypt((char *) buf, buf, (const uchar *) scramble_arg, SCRAMBLE_LENGTH);
  /* now buf supposedly contains hash_stage1: so we can get hash_stage2 */
  mysql_sha1_reset(&sha1_context);
  mysql_sha1_input(&sha1_context, buf, SHA1_HASH_SIZE);
  mysql_sha1_result(&sha1_context, hash_stage2_reassured);
  return memcmp(hash_stage2, hash_stage2_reassured, SHA1_HASH_SIZE);
}

Note the return value is the result of the call to memcmp. The patch just affects that line of code. Here’s what changed:

1
2
-  return memcmp(hash_stage2, hash_stage2_reassured, SHA1_HASH_SIZE);
+  return test(memcmp(hash_stage2, hash_stage2_reassured, SHA1_HASH_SIZE));

Parting thoughts

So you can draw the natural conclusion that if you compiled MySQL with optimizations, you are most likely safe against this vulnerability. It’s always a good idea to upgrade when these kinds of patches are provided, but when the upgrade process can be daunting and possibly cause downtime, it’s nice to know that waiting for the right opportunity is a reasonable decision if you aren’t impacted.

Programming Clojure Released!

| Comments

I’m happy to announce that Programming Clojure 2nd Edition has landed! Working with Stuart and the Prags was a fantastic experience. If you are looking to learn Clojure or want to broaden your knowledge, this book will help you get there. If you have already picked up and read the book, please let me know your thoughts by writing a review. It’s a great feeling to have this done and out there and I look forward to tacking another book down the road.

Craftsman Swap - Day 3

| Comments

After a great first couple of days I was really starting to catch a groove at the Obtiva office. I switched off to a new project and a new pair. The task today was to take a look at integrating Paypal’s express checkout for an online retail application in Rails. My pair for the day was Nate Jackson. Nate started out at Obtiva as an apprentice and became a full time employee after his journey was complete. It’s great to see people who have enjoyed the experience and had such a positive view of their mentorship that they want to continue to work with their mentors. It is also great to see the mentors accept everyone as peers. It might not seem like a big deal, but it makes a huge difference!

One of the things I love about pairing is sitting down for the first time with somebody and observing how they work. Ruby on Rails gives us a common ground for ideas and communication, but the way we communicate is always different. It’s fun to see different tools and environments, and to see in what ways people invest in their tools to help make them a more productive programmer. We started out our pairing by firing up the one true editor and taking a tour of the application to get an idea of what we needed to do to accomplish our goals. The morning seemed to fly by. As soon as we were done walking through the application and reading the PayPal documentation it was time to head to lunch.

After lunch it was time to get cracking on our task. We were off to a great start with PayPal using the active_merchant gem. Unfortunately that didn’t last long. We had setup the basic developer sandbox accounts and got a simple example working, but found out that the proper solution was the “Pro” remix. This involved much more setup with the developer sandbox. The nice part is that since the Obtiva gang works together in the same room, we were able to get the pre-baked account credentials from others that knew where we needed to be. After fighting for a good part of the day around settings in the developer sandbox, and things not working properly, we arrived at what seemed to be a reasonable solution. The bad part was that PayPal just wasn’t cooperating with us. We were able to confirm that things were working with a live account, just not a developer sandbox account.

Nate and I were pleasantly suprised at the end of the day by a phone call from Carson Conant over at Medifly. Palm had contacted him to produce an app for the Pre and Pixi that would be able to stream the “Hope for Haiti” concert and allow for people to donate directly from their phones. The only downside was that we would only have one day to complete it. Obtiva had already built the original Pre app for Mediafly so we at least had a base app to work from. I was excited and nervous all at once for what day four would bring.

Craftsman Swap - Day 2

| Comments

Context and situational awareness are a very important key to working well with others, and day two of my journey solidified that idea. The day started off similar to my first day. I was excited to be working with Ryan on Mad Mimi for another day. Having a better understanding of how Ryan works and how things at Obtiva flow during the day helped me concentrate more on getting real work done. After the morning standup Ryan and I were off to new feature development. We started off by writing tests for importing csv lists of recipients or “audience members” for email campaigns. Ryan had already done a lot of the leg work and refactoring, so it was all about tying a bow around things. Ryan and I were into a good pairing groove and the morning seemed to fly by.

Lunch time brought about a tradition that every company should adopt. Obtiva calls this tradition Geekfest, which is a “lunch and learn” style gathering where someone gives a talk on something new. I know that a lot of people do this informally, but Obtiva has made this a regular event and I applaud the effort that can go into to herding a bunch of developers in a room during the work week. I was also surprised to see the actual number of employees that Obtiva has. If it weren’t for Geekfest I probably wouldn’t have been able to meet all the other developers that work on site during the week. I gave a presentation on compojure, a Sinatra like web framework for clojure.

The day rounded out with Ryan and I making some UI enhancements and finishing up the feature from the morning. Since the deployment strategy involves multiple servers, keeping the csv lists around for batch processing presented an interesting deployment issue. We wrote out a few ideas for deployment solutions and then got in contact with Rails Machine to get their opinion so we could start prepping for the feature release.

Since this was my last day pairing with Ryan we ended the day with a retrospective to talk about how our pairing experience went. A few others sat in to offer outside perspectives and chat about initial impressions and what could be better throughout the rest of the week. This was a great way to end the day and pump me up for what’s to come!

Craftsman Swap - Day 1

| Comments

I started my journey Sunday night to Chicago where I was welcomed into the Obtiva corporate apartment by Ethan Gunderson. I was impressed by the layout and attention to design. It is a swank little downtown pad that anyone would be happy to have. Ethan is one of Obtiva’s apprentices, working for 6 months at Obtiva where he contributes to the team and sharpens his skills as a developer. I really like this idea and would love to see more companies making good developers great.

Monday reminded me of the things I love about being in a big city. The walk to the office in the wonderful Chicago weather made for a nice wake up. The morning started out much like other good agile shops i’ve been to with a morning stand up meeting. A lot of the Obtiva are on site at clients so the meeting was very short. I noticed that after the company wide standup other teams of people had more focused standups which is a great idea. It let’s the group get to work while details are discussed among the relevant parties.

For my first day I got to pair with Ryan Briones. I know Ryan from my days in Columbus OH, so it was nice to start my week of with a familiar face. We started with a brief discussion about the project, Mad Mimi, which is an email marketing system. Ryan had plans laid out for the days work, but as things go we were able to defeat those plans and work on an entirely different set of problems.

We spent the first part of our day working on a little bit of housekeeping and deployment related issues. It was nice to see the server setup and hosting company choices. I also got to see the response and support from Rails Machine first hand as one of our deployments failed. Within a couple of minutes they were on top of everything before anyone could even ask. This kind of reaction to a problem combined with great support staff which they seem to have make for a great hosting experience.

After a great burger at lunch Ryan and I jumped back into things and were greeted with an interesting problem. A company was trying to start an email campaign with a record breaking amount of emails. Things weren’t going quite to plan and it was up to Ryan and I (with some help from Dave Hoover) to track down and fix the problem. After some debugging and extra logging (it’s like cowbell, you always need more), we were able to make some modifications and get the process moving. There are still a few performance tweaks to make, but history was made.

Once things were moving along nicely we moved into some other bug fixes to round out the day. We got to sit down this time and write some tests that proved our bug (and a couple other bugs that weren’t immediately apparent) and get to work. We were able to end the day by getting the bug squashed and watching the record breaking email campaign start it’s mailings.

The evening brought good food and good company as I went to dinner with Ryan, Ethan, and Ryan’s wife Stephanie. We ate some sushi down by Ryan’s house at a place called Ra. We ended the night with some coffee and good conversation that I hope will continue throughout the rest of the week.

Craftsman Swap

| Comments

I am headed to Chicago this week to work at [Obtiva][http://obtiva.com] for the first of what I hope will be many craftsman swaps. I am really looking forward to hanging out with the crew up there and learning how other top notch development companies work. I will be keeping notes of my experiences while I am up there and will report a few times during the week.

Tell RCov to Cov Off

| Comments

Chad and I just released a new version of RCov! This version offers the ability to tell RCov to ignore certain pieces of code. This is useful if you are experiencing an bug in RCov, or if you just plain old disagree with RCov about the coverage status of a particular chunk of code. Here’s an example of implementation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class Foo
  def bar
    1 + 2
  end

  #:nocov:
  def quux
    "s" + "hit"
  end
  #:nocov:
end

require 'test/unit'
class FooTest < Test::Unit::TestCase
  def test_bar
    assert_equal 3, Foo.new.bar
  end
end

Running this will show you 100% code coverage. This feature (for now) is pretty simple and does not do anything more than tell RCov to count the code as covered. If this feature takes off, I will spend more time making it smarter about the code and reporting so that you can easily tell the difference.

RCov Examples

| Comments

Are you interested in trying out RCov? Are you unsure of how to start? Are you looking to improve the RCov-fu on your project? I have just the solution for you! I have started a project that demonstrates different RCov configurations for different testing frameworks. This project is meant to help spread RCov related knowledge. You can get the bits by running:

git clone git://github.com/abedra/rcov-examples.git

Feel free to browse around and submit any feedback. I am hoping to get as many samples in as possible, so feel free to contribute!