Netlink (2.6.8)

Netlink is a socket-based interprocess communication mechanism. User programs can communicate each other with netlink sockets. Programs can use the netlink via the library libnetlink or can directlyinterface to the low-level kernel api. Here i describe how to use the latter.

The netlink "protocol" is message oriented, and the programs must send/receive data encapsulated in netlink messages. The system calls sendmsg.2 and recvmsg.2 are used to send and receive these messages. Their syntax is

Here s is a socket descriptor, obtained with the socket.2 system call, with PF_NETLINK type, and a netlink protocol, a number between 0 and 32. Protocols are listed in the include file linux/netlink.h. A user program can specify a listed protocol or can pick an unused value.

The struct msghdr contains the following fields:

• msg_name, addresses info, and its length msg_namelen. These are used if the socket is not connected/bound to an address;
• msg_iov, an array of struct iovec (each has a pointer, iov_base, to data and their length, iov_len), and the array size msg_iovlen;
• the ancillary data msg_control and their length msg_controllen;
• the message flags msg_flags. This is not used by sendmsg(), which refers to its flags parameter. recvmsg() copies the flags parameter into msg_flags, and may further chnage it depending from the result of the call.

Netlink in user space

To use the netlink, the first entry in the message iovec must be a netlink message header, struct nlmsghdr, which contains

• the total length nlmsg_len of the message, netlink header plus payload;
• the netlink message nlmsg_flags;
• and three fields opaque to the kernel: type, sequence number and pid, nlmsg_type, nlmsg_seq and nlmsg_pid, which can be used freely by the usr program.

To send a message, a program sets the peer netlink sockaddr in the message msg_name, fills in a iovec with first entry a proper netlink message header, and calls sendmsg().

To receive a message, a program either binds the socket to the receiving address or set the peer netlink sockaddr in the message header, and calls recvmsg(). The first mechanism is useful when a server sets up a receiving netlink and advertise the address so that clients can send it their messages. The second mechanims applies when sender and receiver are in a peer-to-peer relation and know each other netlink addresses.

The header file netlink.h defines a few utility macroes. Among the others,

• NLMSG_LENGTH(payload_size) return the proper length to put in nlmsg_len
• NLMSG_SPACE(payload_size) return the number of bytes the netlink message would use
• NLMSG_DATA(nlh) given a pointer to the netlink message, return a pointer to the payload
• NLMSG_PAYLOAD(nlh, len) return the length of the payload associated with the netlink message

After this brief overview of netlink from the user point of view, you should be able to do the following exercises.
Exercise 1. Write a server that advertise a netlink address with its pid, and a client that sends it a message. Display the content of the message by the server to make sure that the proper data are received. Include a payload in the message.
Exercise 2. Write a pair of sender and receiver that communicate on a peer-to-peer basis. Check the content of the message and include a payload.

Here is a sample user program and a makefile. Just do make test to compile two test programs, netlink-cs and netlink-p2p out of the same source. Run netlink-cs with no args to start a server. It prints its pid. Run it again with arg the server pid, to start the client. For netlink-p2p run with no args to start the server (it uses a fixed netlink pid),then run it with anything as arg to start the client (which uses ianother fixed netlink pid).

A library is available for programming with netlink. It is included in ftp://ftp.inr.ac.ru/ip-routing/iproute2*. I have not checked it out yet, so i cannot say about it at the moment.

Netlink in kernel space

The kernel sources for the netlink are in the subdirectory net/netlink.

The netlink supports a fixed number (MAX_LINKS = 32) of "protocols". All the sock's for a given "protocol" are linked in a list. The heads of these lists form the netlink table nl_table array. The table is protected by a lock (nl_table_lock) and nl_table_users counts the users of the table. Functions are provided to

• lock/unlock: netlink_table_grab() and netlink_table_ungrab();
• increase/decrease user count: netlink_lock_table() and netlink_unlock_table();
• lookup a pid in a list: netlink_lookup(protocol, pid);
• insert/remove a sock in the list of its protocol, netlink_insert(sk, pid) and netlink_remove(sk);

The netlink register with the socket layer invoking sock_register() which sets the protocol family (PF_NETLINK) of the netlink, and install the function used to create a netlink instance, netlink_create(). The netlink is removed by unregistering with the socket layer (function sock_unregister()).

netlink_create() takes two parameters: a socket pointer and a netlink protocol number (called "unit"). The protocol must not exceed the maximum protocl number. The socket must be of type either RAW or DGRAM; other types are not supported. Its operations are sets to the netlink operations (see below). A new sock is allocated, and initialized (core/sock.c::sock_init_data()). In particular: its queues (receive, write, and error) are initialized, its sk_socket is set to the parameter socket, ... Furthermore sk_protinfo (netlink options) is allocated (but cleared), sk_destruct is set to netlink_sock_destruct, and sk_protocol is set to the netlink protocol number. Finally it increments the global netlink socket number netlink_sock_nr.

The sock netlink options contain

• the source pid and groups;
• the destination dst_pid and dst_groups;
• the state;
• the wait queue head;
• a spin_lock to protect the callback;
• a netlink callback struct;
• two functions: handler, data_ready

The socket destructor netlink_sock_destruct() purge the receive queue, cheks that the socket has the SOCK_DEAD flag, frees the sk_protinfo, and decrement the global netlink socket number netlink_sock_nr.

The file af_netlink.c defines the netlink operations:

• netlink_release(sock) removes (ie, unlink) the socket' sock from the netlink table, purge the write queue, and "put" the sock.
• netlink_bind(sock, addr, addr_len) binds a netlink socket with a socket address (of type struct sockaddr_nl, ie, with family AF_NETLINK). N.B. addr_len is not used. Only root can listen multicasts. If the the socket netlink options have a pid, this must equal the address nl_pid (in this case the options' groups is copied from the address nl_groups). Otherwise if the address nl_pid is 0 it calls netlink_autobind(). Finally, if the nl_pid is not zero, the sock is inserted in the netlink table of the netlink protocol number (the sock's sk_protocol) (netlink_insert()) with the address nl_pid.
• netlink_connect(sock, addr, alen, flags) connects a netlink socket address to a socket. N.B. alen and flags are not used. The address family must be AF_NETLINK. Only root can connect an address for multicast (ie with nl_groups not 0). If the socket's sock has a non-zero pid netlink options, invokes netlink_autobind() which, briefly, inserts the sock in the protocol list. Finally the sock sk_state is set to NETLINK_CONNECTED, and the destination pid and groups are copied from the address to the netlink options.
• netlink_getname(sock, addr, addr_len, peer) copies the netlink address from the socket's sock in addr. If peer is set the destination pid and groups are copied; otherwise the source ones are copied.
• netlink_sendmsg(kiocb, sock, msg, len) sends a message. The sock_iocb is extracted from the kiocb. Messages with msg_flags MSG_OOB are not supported. [TODO ... scm_cookie scm_send() ]. If the message length len exceeds sk_sndbuf - 32 EMSGSIZE is returned. The netlink address is obtained from the msg msg_name. If the msg_namelen is non-zero (and the address family is AF_NETLINK) the destination pidand groups are copied from the address (only root can do multicast), otherwise they are taken from the socket's sock netlink options. It the options have zero pid, the sock is bound (netlink_autobind()). A socket buffer is allocated for the message (ENOBUFS if allocation fails). The source/destination and the credentials are saved in the socket buffer because the netlink is asynchronous. The data are copied from the message msg_iov, to the socket buffer. [TODO ... security_netlink_send() ]. Finally either netlink_broadcast() or netlink_unicast().
• netlink_recvmsg(kiocb, sock, msg, len, flags) receives a message. The sock_iocb is extracted from the kiocb. The flag MSG_OOB is not supported. The flag MSG_DONTWAIT is supported. Calls the skb_recv_datagram(). If the received bytes are less then len the message msg_flags MSG_TRUNC is set. The received bytes are copied to the message msg_iov. The message msg_name is written with the netlink family, AF_NETLINK, and the pid and groups from the received socket buffer. [TODO ... the scm_cookie scm_recv() ]. The buffer is freed, and waiting process waken (netlink_rcv_wake()).

• sock_no_socketpair
• sock_no_accept
• sock_no_ioctl
• sock_no_listen
• sock_no_shutdown
• sock_no_setsockopt
• sock_no_getsockopt
• sock_no_mmap
• sock_no_sendpage

Netlink kernel programming

Netlink can be used also as a user-kernel communication channel. A user program sends and receive a netlink message to the kernel by specifying a pid 0.

It is necessary that a kernel component (a module) creates a netlink unit ("protocol") with a input handling function. The source in af_netlink.c exports the function netlink_kernel_create(unit, input) which takes the unit number, and the "input" callback function. This function is invoked when a netlink message arrives for this unit. It has signature

It should dequeue socket buffers from the sk_receive_queue of sk, and process the netlink message contained in their data. The socket buffer should be freed afterwards.

The input function is called in the context of sendmsg (of the sending process). If the message processing is slow, it is better to defer it to a kernel thread, and use input to wake up the thread.

The kernel module sends netlink messages to user programs calling the functions

The first is used for sending a message to a single process, the second for multicasting. The sk parameter is the netlink sock returned by netlink_kernel_create. The socket buffer skb should contain the message data (netlink header plus payload), and pid is the receiving process pid. For broadcast pid is zero and group is a bitmask of the ORed receiving groups.

The flag nonblock corresponds to MSG_DONTWAIT. If it is set and no process is ready to receive the message the function returns an error. An error is returned also if there is no process attached to the netlink with that pid.

The kernel writes the source and destination data with the macro NETLINK_CB( skb ), provided in netlink.h, which returns the address of the socket buffer control block (cast to a pointer to netlink_skb_parms). The important fields to set are

• pid for the source pid (0 for the kernel);
• groups for the source groups;
• dst_pid for the destination pid, ie, the netlink "pid" of the receiving process;
• dst_groups for the destination groups;

When the kernel module is removed its netlink socket should be removed. Netlink does not export the function netlink_release(), but the module can call sock_release() with the netlink socket contained in the field sk_socket of the kernel netlink sock obtained with netlink_kernel_create. This function (see net/socket.c) besides calling netlink_release decreases the counter socket_in_use (and if the socket has a file iput-s it).

Exercise 1
Write a kernel module that creates a netlink and receive messages from user programs and echo then back.

Install the kernel module

You can see the module with

lsmod | grep knetlink

$ cat /proc/net/netlink
sk       Eth Pid    Groups   Rmem     Wmem     Dump     Locks
09fdbe00 0   0      00000000 0        0        00000000 2
...
09fdb800 17  0      00000000 0        0        00000000 2

The module is removed with the command

rmmod knetlink

Here you can find the kernel module and a test program. Compile them with the same makefile used for the user-level sample programs. make, with the target, compiles the kernel module knetlink.ko. make test compiles the user test program netlink-u2k. Install the kernel module, and run netlink-u2k.

Marco Corvi - 2005