Enabling ModSecurity in the Kubernetes Ingress-NGINX Controller

The Kubernetes Ingress-Nginx controller allows us to enable ModSecurity, an OpenSource Web Application Firewall. In this post I will go over how to use and configure ModSecurity with Ingress-Nginx.

“Marshmello walking down the side of the road with camera on the top of the building” by arvin febry on Unsplash

What is ModSecurity

ModSecurity is a web application firewall which is used to protect your web application from a variety of attacks.

In Ingress-NGINX you will be able to ModSecurity in a variety of ways:

  • Default Configuration
  • OWASP Core Rule Set
  • Snippet

Default Configuration

When enabling ModSecurity without the OWASP Core Rule Set or a Snippet, a Default Configuration from ModSecurity is used. It runs only in “Detection Only Mode” and is non-disruptive to your applications.

Usually you would run in “Detection Only Mode” for some time and review all the events generated by ModSecurity. Then you should start fine tuning it towards your specific application.

The default Configuration also contains different rules for ModSecurity which are heavily documented which you can review in the link above. It is recommended that you modify this file as needed.

OWASP Core Rule Set

The OWASP Core Rule Set is a set of generic attack detection rules that can be used with ModSecurity. It provides protection from a variety attacks, while limiting false positives.

It protects against the OWASP Top 10 and many others. Some of the common attacks it protects against are:

  • SQL Injection (SQLi): SQL Injection is an attack in which an SQL query is inserted into from client to application.
  • Cross Site Scripting (XSS): Cross-site scripting attacks occur when an attacker uses a web application to send malicious code to a different end user.
  • Local File Inclusion (LFI): Local File Inclusion is the process of including files, that are already locally present on the server, through the exploiting of vulnerable inclusion procedures implemented in the application.
  • Remote File Inclusion (RFI): Remote File Inclusion is the process of including remote files through the exploiting of vulnerable inclusion procedures implemented in the application.
  • Remote Code Execution (RCE): Code Injection is the general term for attack types which consist of injecting code that is then interpreted/executed by the application.
  • Session Fixation: Session Fixation is an attack that permits an attacker to hijack a valid user session.
  • Scanner Detection: Scanners can be used by hackers for malicious reasons. If you aren’t expecting any scan on your system, it is useful to detect that it is happening.
  • Metadata/Error Leakages: Leaking of descriptive errors or sensitive information.

Note, that the OWASP Core Rule Set configuration is highly documented and it is recommended that it be tuned.

Snippet

Using a ModSecurity Snippet, you can insert any set of ModSecurity Directives Desired. For a list of directives you can checkout the ModSecurity documentation.

I recommended once you have ModSecurity configuration tuned with all of your needs you use the ModSecurityConfig directive to point toward your configuration.

In order to add a custom ModSecurity configuration, you can either add all the required code to the Snippet(maybe hard to maintain) or build an Ingress-NGINX image containing your ModSecurity configuration. Search for “build modsecurity library” in the nginx build file for an example.

Note: Customizing and Building NGINX with Custom ModSecurity files is out of the scope of this blog post, but there is some documentation here.

Setting Up the Ingress Controller

There are many ways of configuring Ingress-Nginx on your Kubernetes cluster. If you already have an Ingress-Nginx controller setup, then you can skip this step, However….

Note that this guide was written using Minikube version 0.31 running Ingress-Nginx version 0.21. If you already have Minikube installed, you may need to update.

Deploying an Application with ModSecurity

  1. Deploy our pods(containers), using deployments. A deployment, pretty much just manages the state of our pods, It’s out of the scope of this tutorial, but you can learn more here
$ echo "
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: meow
spec:
replicas: 2
selector:
matchLabels:
app: meow
template:
metadata:
labels:
app: meow
spec:
containers:
- name: meow
image: gcr.io/kubernetes-e2e-test-images/echoserver:2.1
ports:
- containerPort: 8080
" | kubectl apply -f -
# wait a min for the deployment to be created
$ kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE
meow 2 2 2 2
# you should have 2 pods running
$ kubectl get pods
NAME READY STATUS
meow-5557bc7c54-cw2ck 1/1 Running
meow-5557bc7c54-kfzm5 1/1 Running

gcr.io/kubernetes-e2e-test-images/echoserver:2.1 just responds with information about the request.

2. Expose our pods using Services. More info about exposure here

$ echo "
apiVersion: v1
kind: Service
metadata:
name: meow-svc
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: meow
" | kubectl apply -f -
# wait a min for the service to be created
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
meow-svc ClusterIP 10.107.78.24 <none> 80/TCP

This allows the meow pods, created with the deployment above to be accessed from within the cluster. That way we can access the meow pods from the nginx-ingress-controller pod.

3. Setup the Ingress Rules, enabling ModSecurity with a custom snippet

$ echo "
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/enable-modsecurity: \"true\"
nginx.ingress.kubernetes.io/modsecurity-snippet: |
SecRuleEngine On
SecRequestBodyAccess On
SecAuditEngine RelevantOnly
SecAuditLogParts ABIJDEFHZ
SecAuditLog /var/log/modsec_audit.log
SecRule REQUEST_HEADERS:User-Agent \"fern-scanner\" \"log,deny,id:107,status:403,msg:\'Fern Scanner Identified\'\"
name: meow-ingress
namespace: default
spec:
rules:
- http:
paths:
- backend:
serviceName: meow-svc
servicePort: 80
path: /meow
" | kubectl apply -f -
$ kubectl get ing meow-ingress
NAME HOSTS ADDRESS PORTS
meow-ingress * 10.0.2.15 80

This ingress resource enables ModSecurity with custom rules for testing purposes. You can read more about these directives and many others in the ModSecurity documentation.

Testing it out

Now that we have ModSecurity configured, we can send some requests and examine the ModSecurity logs.

$ minikube ip
192.168.99.100
# Here you can see the the meow application response
$ curl https://192.168.99.100/meow -k
...
Request Headers:
accept=*/*
host=192.168.99.100
user-agent=curl/7.54.0
...
# This path is forbidden because of the SecRule for blocking any request containing the header "user-agent: fern-scanner"
$ curl https://192.168.99.100/meow -k -H "user-agent: fern-scanner"
...
<center><h1>403 Forbidden</h1></center>
...

We can also take a look at the audit-logs:

# The AuditLog will contain only warnings and errors since the SecAuditEngine directive is set to RelevantOnly, It will also contain the parts specified in SecAuditLogParts$ kubectl get pods -n kube-system | grep nginx
nginx-ingress-controller-6766d5b6c7-9qvfm
$ kubectl exec -it -n kube-system nginx-ingress-controller-6766d5b6c7-9qvfm cat /var/log/modsec_audit.log
---kRdKBbup---A--
[10/Dec/2018:23:32:08 +0000] 154448472869.334297 192.168.99.1 50447 192.168.99.1 443
---kRdKBbup---B--
GET /meow HTTP/2.0
host: 192.168.99.100
accept: */*
user-agent: fern-scanner
...
---kRdKBbup---E--
<html>\x0d\x0a<head><title>403 Forbidden</title></head>\x0d\x0a<body>\x0d\x0a<center><h1>403 Forbidden</h1></center>\x0d\x0a<hr><center>nginx/1.15.6</center>\x0d\x0a</body>\x0d\x0a</html>\x0d\x0a
---kRdKBbup---F--
HTTP/2.0 403
Server: nginx/1.15.6
Date: Mon, 10 Dec 2018 23:32:08 GMT
Content-Length: 153
Content-Type: text/html
Connection: close
---kRdKBbup---H--
ModSecurity: Access denied with code 403 (phase 1). Matched "Operator `Rx' with parameter `fern-scanner' against variable `REQUEST_HEADERS:user-agent' (Value: `fern-scanner' ) [file "<<reference missing or not informed>>"] [line "7"] [id "107"] [rev ""] [msg "Fern Scanner Identified"] [data ""] [severity "0"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "192.168.99.1"] [uri "/meow"] [unique_id "154448472869.334297"] [ref "o0,12v64,12"]
...

Troubleshooting

If you are having issues with setting up ModSecurity or some ModSecurity Directives, there are a few ways of troubleshooting:

  1. Examine the Ingress-Nginx Logs:
# Note: I added "Ahhhh Ok" Somewhere in the ModSecurity Snippet that would cause an issue$ kubectl get pods -n kube-system | grep nginx
nginx-ingress-controller-6766d5b6c7-9qvfm
$ kubectl logs -n kube-system nginx-ingress-controller-6766d5b6c7-9qvfm
...
Error: exit status 1
2018/12/11 03:56:03 [notice] 7723#7723: ModSecurity-nginx v1.0.0
2018/12/11 03:56:03 [emerg] 7723#7723: "modsecurity_rules" directive Rules error. File: <<reference missing or not informed>>. Line: 3. Column: 12. Invalid input: Ahhhh Ok in /tmp/nginx-cfg303221100:331
nginx: [emerg] "modsecurity_rules" directive Rules error. File: <<reference missing or not informed>>. Line: 3. Column: 12. Invalid input: Ahhhh Ok in /tmp/nginx-cfg303221100:331
nginx: configuration file /tmp/nginx-cfg303221100 test failed
...

2. Examine the Nginx configuration which presented an error:

# Check the temporary config generated, around the line(331) stated in the error in step 1
$ kubectl exec -it -n kube-system nginx-ingress-controller-6766d5b6c7-9qvfm -- sed -n '325,335p' /tmp/nginx-cfg303221100
# After resolving the issue you can examine the configuration to see if your changes took effect
$ kubectl exec -it -n kube-system nginx-ingress-controller-6766d5b6c7-9qvfm cat /etc/nginx/nginx.conf
# Configuration checksum: 2601299549274808909
...

Annotations

All the annotations pertaining to Modsecurity are as follows:

  • nginx.ingress.kubernetes.io/enable-modsecurity: This must be set to true in order to enable ModSecurity. If enable-owasp-core-rules or a modsecurity-snippet annotations are not present, then ModSecurity will run in “Detection-Only” mode using the recommended configuration.
  • nginx.ingress.kubernetes.io/enable-owasp-core-rules: Enables the OWASP Core Rule Set. Note, it should not be used with the modsecurity-snippet annotation.
  • nginx.ingress.kubernetes.io/modsecurity-snippet: Allows you to add your own set of modsecurity rules via a snippet. Note, it should not be used with the enable-owasp-core-rules annotation.
  • nginx.ingress.kubernetes.io/modsecurity-transaction-id: Allows you to pass transactionIDs from nginx to ModSecurity.

For more detailed information on all the annotation usage see the official documentation.

Further Reading

NGINX-Ingress has some ModSecurity configurations out of the box, however, in many cases there must be fine tuned rules for your specific needs.

The following are great resources to get started:

Thanks for Reading and I hope you enjoyed 😃

Senior Technical Marketing @ GitLab 🦊, Developer @ home, Keeping ATX Weird 😜