Nginx Internals

Joshua Zhu 09/19/2009

Agenda

Source code layout Key concepts and infrastructure The event-driven architecture HTTP request handling Mail proxying process Nginx module development Misc. topics

Source Code Layout

Files

$ find . -name "*\.[hc]" -print | wc l 234 $ ls src core event http mail misc os $ find . -name "*\.[hc]" -print | xargs wc -l | tail -n1 110953 total

Lines of code

Code Organization

core/

The backbone and infrastructure The event-driven engine and modules The HTTP server and modules The Mail proxy server and modules C++ compatibility test and the Google perftools module OS dependent implementation files

event/

http/

mail/

misc/

os/

Nginx Architecture

Non-blocking Event driven Single threaded[*] One master process and several worker processes Resource efficient Highly modular

The Big Picture

Agenda

Source code layout Key concepts and infrastructure The event-driven architecture HTTP request handling Mail proxying process Nginx module development Misc. topics

Memory Pool

Avoid memory fragmentation Avoid memory leak Allocation and deallocation can be very fast Lifetime and pool size

Cycle Connection Request

Memory Pool (contd)

ngx_pool_t

Small blocks Large blocks Free chain list Cleanup handler list ngx_palloc

API

memory aligned

ngx_pnalloc ngx_pcalloc

Memory Pool Example (1 Chunk)

Memory Pool Example (2 Chunks)

Buffer Management

Buffer

Pointers

memory

start/pos/last/end file_pos/file_last/file

file

Flags

last_buf last_in_chain flush in_file memory

Buffer Management (contd)

Buffer chain

Singly-linked list of buffers Context

Output chain

in/free/busy chains

Output filter Writer context

Chain writer

String Utilities

ngx_str_t

data len sizeof() - 1

Memory related String formatting String comparison String search Base64 encoding/decoding URI escaping/unescaping UTF-8 decoding String-to-number conversion

Data Structures

Abstract data types

Array List Queue Hash table Red black tree Radix tree Set object values after added

Characteristic

keep interfaces clean efficient

Chunked memory (part)

Logging

Error log

Level Debug Multiple logs Log format

Access log

variables

Per location

Rotation

Configuration File

Directive

name type set conf offset post ngx_conf_parse init merge

Parsing

Values

Configuration File (contd)

Block

events http server upstream location if Buildins Other types

Variables

http_ sent_http_ upstream_http_ cookie_ arg_

Agenda

Source code layout Key concepts and infrastructure The event-driven architecture HTTP request handling Mail proxying process Nginx module development Misc. topics

Master and Workers

Master

Monitor workers, respawn when a worker dies Handle signals and notify workers

exit reconfiguration update log rotation

Worker

Process client requests

handle connections

Get cmd from master

Master Process Cycle

Worker Process Cycle

Inter-process Communication

Signals

Channel

socketpair command

Shared memory

Connection counter Stat Atomic & spinlock Mutex

Event

ngx_event_t

Read Write Timeout

Callbacks Handlers

ngx_event_accept ngx_process_events_and_timers ngx_handle_read_event ngx_handle_write_event Posted accept events queue Posted events queue

Posted events

Time Cache

The overhead of gettimeofday() Time cache variables

ngx_cached_time ngx_current_msec Time strings

ngx_cached_err_log_time ngx_cached_http_time ngx_cached_http_log_time

Timer resolution

Interval timer

setitimer()

Events and Timers Processing

Timer Management

Actions

Add a timer Delete a timer Get the minimum timer O(log n) complexity

Red black tree[*]

Accept Mutex

Thundering herd Serialize accept() Lock/unlock Listening sockets Delay

I/O

Multiplexing

kqueue/epoll

NGX_USE_CLEAR_EVENT (edge triggered) NGX_USE_LEVEL_EVENT (level triggered)

select/poll/dev/poll

sendfile() writev() direct I/O mmap() AIO TCP/IP options

Advanced I/O

TCP_CORK/TCP_NODELAY/TCP_DEFER_ACCEPT

Agenda

Source code layout Key concepts and infrastructure The event-driven architecture HTTP request handling Mail proxying process Nginx module development Misc. topics

Important Structures

Connection

ngx_connection_t ngx_http_connection_t ngx_http_request_t

HTTP connection

HTTP request

headers_in headers_out

Virtual Servers

Address Port Server names Core server conf

Locations

Location tree

Static Regex

= ^~ ~ ~*

Per-location configuration

Value

inheritance override

Handler try_files/post_action/error_page

Named location

HTTP Contexts

Types

main_conf srv_conf loc_conf ngx_http_get_module_main_conf ngx_http_get_module_srv_conf ngx_http_get_module_loc_conf ngx_http_conf_get_module_main_conf ngx_http_conf_get_module_srv_conf ngx_http_conf_get_module_loc_conf ngx_http_get_module_ctx ngx_http_set_ctx

Request

Parse conf file

Module context

HTTP Handling

Receive data Parse the request Find the virtual server Find the location Run phase handlers Generate the response Filter response headers Filter the response body Send out the output to the client

Request Parsing

Request line Headers Interesting tricks

Finite state machine ngx_strX_cmp

Phases and Handlers

Phases

POST_READ SERVER_REWRITE FIND_CONFIG REWRITE POST_REWRITE PREACCESS ACCESS POST_ACCESS TRY_FILES CONTENT LOG Checker Handler Next

Phase handler

Phases and Handlers (contd)

Phase engine

Handlers server_rewrite_index location_rewrite_index r->phase_handler ngx_http_core_generic_phase ngx_http_core_find_config_phase ngx_http_core_post_rewrite_phase ngx_http_core_access_phase ngx_http_core_post_access_phase ngx_http_core_try_files_phase ngx_http_core_content_phase

Default checkers

Phases and Handlers (contd)

phase POST_READ SERVER_REWRITE REWRITE PREACCESS ACCESS CONTENT LOG modules realip rewrite rewrite limit_req, limit_zone, realip access, auth_basic autoindex, dav, gzip, index, random_index, static log

Filter Chain

Singly-linked list like (CoR) Filter response only

Header filter Body filter ngx_http_send_header

Send out the response

top_header_filter ngx_http_top_body_filter

ngx_http_output_filter

ngx_http_header_filter ngx_http_copy_filter ngx_http_write_filter

Process order

Filter Chain Example

HTTP Handling Example

curl -i http://localhost/

HTTP Keep-Alive

Request memory reuse Connection memory shrink Keep-alive timeout Request count

Subrequest

Filters

Addition filter SSI filter

Maximum subrequests

Internal Redirect

Return a different URL than originally requested Examples

try_files index/random_index post_action send_error_page upstream_process_headers

Upstream

Hooks

input_filter_init input_filter create_request reinit_request process_header abort_request finalize_request rewrite_redirect FastCGI Proxy Memcached

Modules

Event pipe Load balancer

Agenda

Source code layout Key concepts and infrastructure The event-driven architecture HTTP request handling Mail proxying process Nginx module development Misc. topics

Mail Proxy

Sequence diagram

Mail Proxy (contd)

Mail session

Command parsing Packets relay Load balancing Authentication rewriting Black lists/white lists

Things you can do

Agenda

Source code layout Key concepts and infrastructure The event-driven architecture HTTP request handling Mail proxying process Nginx module development Misc. topics

General Module Interface

Context

index & ctx_index

Directives Type

core/event/http/mail init_master

Hooks

called at master process initialization called when the module is loaded called at worker process initialization called at worker process termination called at master process termination

init_module

init_process

exit_process

exit_master

Core Module Interface

Name Hooks

create_conf init_conf Core Events Log HTTP

Examples

Event Module Interface

Name Hooks

create_conf init_conf event_actions

add del enable disable add_conn del_conn process_changes process_events init done

Mail Module Interface

Protocol

type init_session init_protocol parse_command auth_state

create_main_conf init_main_conf create_srv_conf merge_srv_conf

HTTP Module Interface

Hooks

preconfiguration postconfiguration create_main_conf init_main_conf create_srv_conf merge_srv_conf create_loc_conf merge_loc_conf

A Hello World HTTP Module

Creating a hello world! module

Files

ngx_http_hello_module.c config ./configure add-module=/path/to/hello/module location & directive

Build

Configuration

Agenda

Source code layout Key concepts and infrastructure The event-driven architecture HTTP request handling Mail proxying process Nginx module development Misc. topics

Auto Scripts

Handle the differences

OS Compiler Data types Libraries

Module enable/disable Modules order

Reconfiguration

Hot Code Swapping

Thank You!

My site: http://www.zhuzhaoyuan.com My blog: http://blog.zhuzhaoyuan.com