WAF code analysis: openresty

Category: Tags: ,

Why are there such series of articles?

Since the vigorous development of information security in recent years, the construction of information security has gradually developed. As a security engineer of Party A, a personal security department, using open source WAF to deploy defense attacks seems to be a very common thing, but open source The downside is that no one can provide technical support in time. If there is a problem, you can only maintain it yourself. I believe that maintaining WAF code and rules is a very energy-consuming task. It is better to use commercial incense. If you don’t have a safe budget, you might as well join me in the world of WAF development.

Previously, the WAF maintenance and loading new functions were only processed in the access phase of openresty, and there was no overall understanding of how the entire process works. The next article will focus on the openresty processing phase, WAF code writing and other content. Some of the content may seem verbose, because this is the result of my thinking about technical details.

Why can openresty become the core choice for WAF development? For business systems, nginx is generally placed on the outermost layer, as load balancing, forwarding traffic, and the official modsecurity rules are also provided for WAF defense attacks, but those who have used it also know that the code is written in C, but The writeability is poor and the defense rules are not easy to maintain. The most important point is performance loss. I wrote a WAF performance test report before. Although there may be scene deviations, it is obvious that the performance of nginx is not as good as openresty. And one important thing is that the nature of openresty is still nginx, but the lua script language is added for embedding, performance processing is improved and complex processing scenarios can be written.

These WAFs are mainly popular in the open source community: jxwaf, openstar, ngx_lua_waf (listed in no particular order). I will mainly analyze the code of jxwaf here. Of course, I will find other WAF codes for ideas collision and learning.

Openresty processing flow

Openresty has 11 processing stages, as shown in the figure below

The scope and function of the specific stages are as follows, source address: https://openresty-reference.readthedocs.io/en/latest/Directives/

stage Scope(nginx.conf) function
init_by_lua* http Initialize nginx and preload lua (executed when nginx starts and reload)
init_worker_by_lua* http Each worker process (worker_processes) is executed when it is created, and is used to start some timing tasks,
For example, heartbeat check, health check of back-end services, regular pull server configuration, etc.;
ssl_certificate_by_lua* server Handling of HTTPS requests
set_by_lua* server, server if, location, location if Process branch processing judgment variable initialization
rewrite_by_lua* http, server, location, location if Forwarding, redirection, caching and other functions
access_by_lua* http, server, location, location if Content processing (WAF rule processing)
content_by_lua* location, location if Content generation, equivalent to response
balancer_by_lua* upstream Load balancing
header_filter_by_lua* http, server, location, location if Process the response headers
body_filter_by_lua* http, server, location, location if Process the response body
log_by_lua* http, server, location, location if Logging

In order to let everyone know these stages more intuitively, I use the nginx configuration file of jxwaf to explain. Of course, friends who don’t understand should go to self-tuition.

http {
    include       mime.types;
    default_type  application/octet-stream;

    client_body_buffer_size  100m;
    client_max_body_size 10m;
    sendfile        on;
    #tcp_nopush     on;
	resolver  114.114.114.114;
  resolver_timeout 5s;
    #keepalive_timeout  0;
    keepalive_timeout  65;
    lua_ssl_trusted_certificate  /etc/pki/tls/certs/ca-bundle.crt;
    lua_ssl_verify_depth 3;
lua_shared_dict limit_req 100m;
lua_shared_dict limit_req_count 100m;
lua_shared_dict limit_attack_ip 100m;
lua_shared_dict limit_bot 100m;
lua_shared_dict waf_common_conf 100m;
lua_shared_dict black_attack_ip 100m;
init_by_lua_file /opt/jxwaf/lualib/resty/jxwaf/init.lua;
init_worker_by_lua_file /opt/jxwaf/lualib/resty/jxwaf/init_worker.lua;
rewrite_by_lua_file /opt/jxwaf/lualib/resty/jxwaf/rewrite.lua;
access_by_lua_file /opt/jxwaf/lualib/resty/jxwaf/access.lua;
header_filter_by_lua_file /opt/jxwaf/lualib/resty/jxwaf/header_filter.lua;
#body_filter_by_lua_file /opt/jxwaf/lualib/resty/jxwaf/body_filter.lua;
log_by_lua_file /opt/jxwaf/lualib/resty/jxwaf/log.lua;
rewrite_by_lua_no_postpone on;
    #gzip  on;
	upstream jxwaf {
	server www.jxwaf.com;
  balancer_by_lua_file /opt/jxwaf/lualib/resty/jxwaf/balancer.lua;
}
lua_code_cache on;
    server {
        listen       80;
        server_name  localhost;
        set $proxy_pass_https_flag "false";
        location / {
            #root   html;
           # index  index.html index.htm;
          if ($proxy_pass_https_flag = "true"){
            proxy_pass https://jxwaf;
          }
          if ($proxy_pass_https_flag = "false"){
            proxy_pass http://jxwaf;
          }
           proxy_set_header Host  $http_host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }

    server {
        listen       443 ssl;
        server_name  localhost;
        ssl_certificate      full_chain.pem;
        ssl_certificate_key  private.key;
        ssl_session_cache    shared:SSL:1m;
        ssl_session_timeout  5m;
        ssl_session_tickets off;
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:ECDHE-RSA-AES128-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA128:DHE-RSA-AES128-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA128:ECDHE-RSA-AES128-SHA384:ECDHE-RSA-AES128-SHA128:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA128:DHE-RSA-AES128-SHA128:DHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA384:AES128-GCM-SHA128:AES128-SHA128:AES128-SHA128:AES128-SHA:AES128-SHA:DES-CBC3-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!R";
        ssl_prefer_server_ciphers  on;
        ssl_certificate_by_lua_file /opt/jxwaf/lualib/resty/jxwaf/ssl.lua;
        set $proxy_pass_https_flag "false";
        location / {
            root   html;
            index  index.html index.htm;
          if ($proxy_pass_https_flag = "true"){
            proxy_pass https://jxwaf;
          }
          if ($proxy_pass_https_flag = "false"){
            proxy_pass http://jxwaf;
          }
            proxy_set_header Host  $http_host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }
    }

}

Explain the attributes of openresty:

lua_ssl_trusted_certificate/etc/pki/tls/certs/ca-bundle.crt;

CA bundle is a file containing root certificates and intermediate certificates. 
Use temsock: sslhandshake method to specify a file path with a trusted CA certificate in PEM format,
which is used to verify the certificate of the SSL/TLS server.

lua_ssl_verify_depth3;

Set the verification depth in the server certificate chain. 
The certificate chain is that the Root CA issues the second-level Intermediate CA,
 the second-level Intermediate CA can issue the third-level Intermediate CA, 
or directly issue the user certificate. A chain of trust is formed from the Root CA to the user certificate: 
If you trust the Root CA, you should trust the second-level Intermediate CA it trusts, 
and you should trust the third-level Intermediate CA until you trust the user certificate. 
The verification depth is set to 3, which means that if the third-level certificate under the Root CA is trusted,
 it is verified.

lua_shared_dict limit_req 100m;

Declare a shared memory area,
 which contains variable dictionary data. 
When nginx starts, it reads the data of dictionary variables.
 The advantage of this is that, for example,
 there are many detection rules, SQL injection,
 XSS, command injection and other rules can be put into memory in advance, 
openresty can go directly to the memory data, instead of opening a file and then closing a packet, 
reducing Read and write to the disk.

rewrite_by_lua_no_postpone on;

Whether to let "rewrite_by_lua" be executed at the end of the rewrite stage, the default value is off,
 that is, the Lua code in "rewrite_by_lua" will be executed after other Nginx rewrite function modules.

lua_code_cache on;

During development and debugging, it can be turned off, because it can be updated in real time, 
and nginx does not need to be restarted. It must be turned on in production, otherwise the performance loss will be very large. .

Counting the 11 execution stages of openresty, jxwaf uses 8 stages. The following describes the code of these 8 stages will be the focus of this series.

Installation configuration
According to the current openresty version required by jxwaf is 1.15.8.3, use the centos7 system to install:

yum install -y readline-devel pcre pcre-devel openssl openssl-devel gcc curl GeoIP-devel  wget 
wget https://openresty.org/download/openresty-1.15.8.3.tar.gz 
tar -xvf openresty-1.15.8.3.tar.gz 
cd openresty-1.15.8.3 
./configure -j2 
make -j2 
make install

Or you can download the openresty version of Windows for just windows system

https://openresty.org/download/openresty-1.15.8.3-win64.zip

The WAF is developed using the LUA programming language. LUA is still a niche language. There is no complete IDE like pycharm. The choice of LUA IDE depends on your needs. You can use vscode+ plug-ins for development.

Lua plugin for syntax highlighting and syntax description

Lua Debug is used to debug and run lua code

Press F5 to start debugging, you can see the code that lua runs successfully

Test

In the nginx.conf configuration file, I turned off lua_code_cache. This is because when testing the code, nginx does not need to be reloaded, nginx will be directly updated, and then I call the test.lua file output content in the content_by_lua_file stage

worker_processes  1;
events {
    worker_connections  1024;
}
http {
lua_code_cache off;
    server {
        location /test {
           default_type 'text/plain';
           content_by_lua_file 'D:/waf/test.lua';
        }
    }
}
#test.lua
local name =  "Anonymous"
ngx.say("Hello, ", name, "!")
ngx.say("test")

The effect is as follows

 

 

 

PS:lua_code_cache enable or disable caching

Code introduction
This parameter is very important for the debugging and development of Lua. Its function is to turn off or turn on the code cache of Lua. If you need to see the effect of code changes in real time when you develop, just turn off lua_code_cache.

grammar Default state Scope
lua_code_cache on | off lua_code_cache on http, server, location, location if

lua_code_cache is turned on by default. When debugging, you can set lua_code_cache off to turn off the cache.

Starting from version 0.9.3, when lua_code_cache is closed, each request of the ngx_lua service will run in a separate Lua VM instance. Therefore, Lua files referenced in set_by_lua_file, content_by_lua_file, access_by_lua_file, etc. will not be cached, and all Lua modules used will be reused. With this, developers can adopt editing and refreshing methods.

But please note that the Lua code written inline in nginx.conf (such as the code specified by set_by_lua, content_by_lua, access_by_lua and rewrite_by_lua) will not be updated when you edit the inline Lua code in the nginx.conf file, because only Nginx The configuration file parser can correctly parse the nginx.conf file. The only way is to reload the configuration file by sending a HUP signal, or just restart Nginx.

Even if code caching is enabled, Lua files loaded by dofile or loadfile in *_by_lua_file cannot be cached (unless you cache the results yourself). Generally, you can use init_by_lua or init_by_lua_file instructions to load all such files, or you can just make these Lua files into real Lua modules and load them through require.

The gx_lua module does not support the statistical mode available with the Apache mod_lua module (yet).

It is strongly recommended to disable Lua code caching for production use and only use it during development, as it will have a significant negative impact on overall performance. For example, after disabling the Lua code cache, the performance of the “hello world” Lua example may drop by an order of magnitude.

 

Reviews

There are no reviews yet.

Be the first to review “WAF code analysis: openresty”

Your email address will not be published. Required fields are marked *