Ideas and methods for realizing real-time log reporting with Nginx pure configuration

Preface

Nginx is a commonly used load balancing gateway. It generates a lot of logs. However, since the Nginx configuration file is a declarative programming paradigm, it is not convenient to describe process control, so log reporting cannot be achieved through simple instructions.

Usually, Nginx log reporting requires writing a shell script or a script in another language to periodically parse the Nginx log file and then report it.

Using the NJS module, real-time log reporting can be achieved.

However, due to the limitation of the instructions supported by the NJS module, it is not possible to implement log reporting well through a single instruction. The combination of multiple instructions can achieve non-blocking real-time log reporting.

This solution is implemented in Nginx and does not rely on other processes such as Node, Python, etc.

Implementation ideas

Nginx has many instructions. The following is a recently explored implementation method. If you have a more elegant implementation method, please leave a message for communication.

Although we have the powerful Njs module to write JS scripts, the instructions of the NJS module have many limitations and cannot achieve any functions like Node.

To achieve real-time reporting of logs, the following two capabilities need to be met:

1. Each request can trigger
2. Report in the background, without blocking the processing of the current request

The commonly used js_set instruction can be triggered on every request, but it only supports synchronous operations. It cannot use fetch and subrequest methods.

The fetch function can be used in the js_content directive. But it can only be used in location. Therefore, other directives can be used to forward the request to the js_content path, and the log report can be completed in the directive.

The auth_request instruction of the http_auth_request_module module is used to perform permission verification of requests, such as jwt verification. This instruction is triggered for each request, creates a subrequest, and determines the result of the permission verification based on the return result of the request.

Therefore, these two modules can be combined to realize log reporting.

Implementation steps

1. Compile Nginx

To implement this function, Nginx needs to support the ngx_http_js_module and ngx_http_auth_request_module modules. These two modules are not installed by default. You need to compile and implement them yourself.

1. For NJS module installation, please refer to When JS Meets Nginx
2. The http_auth_request_module module can be compiled by adding the parameter --with-http_auth_request_module

Compile

./configure --add-module=[NJS module path]/NJS/nginx --with-http_auth_request_module 

make && make install 

2. The configuration file is as follows

http {
    js_import http.js; # Import js file server {
        listen 80;
    
        auth_request /proxy_report; # This instruction is triggered at the beginning of each request, creating a subrequest forwarded to the proxy_report path location / { 
            index index.html index.htm;
        }
        
        location /proxy_report {
            internal; #Limit only internal requests #Save the uri and method data of the original request in the header. Because the auth_request request will modify these data.            
            proxy_set_header X-Original-URI $request_uri;
            proxy_set_header X-Original-METHOD $request_method;
            # Forward to another server proxy_pass http://localhost:8080/report;
        }
    }
    
    server {
        listen 8080;
        # The reporting interface is placed in another server, and there is no auth_request instruction in the server to avoid loop triggering requests location /report {
            #Introduce a js processing script through the js_content instruction to complete the reporting operation js_content http.report;
        }
    }
}

// http.js fileimport qs from "querystring";

async function report (r) {
    let args = {
        // Get the original uri and method from the header uri: r.headersIn['X-Original-URI'],
        method: r.headersIn['X-Original-METHOD'],
        remoteAddress: r.remoteAddress,
        status: r.status,
        headersIn: JSON.stringifry(r.headersIn),
    }
    // Issue an asynchronous request without blocking the current request process and complete the report in the background ngx.fetch(`http://[report service path]?${qs.stringify(args)}`, {
        method: 'GET',
    })
    // Return a status code of 200 to verify the command is successful r.return(200)
}

export default { report }

Summarize

This concludes this article about the ideas and methods of implementing real-time log reporting with Nginx pure configuration. For more relevant content on real-time reporting of Nginx logs, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!