Documentation for Exim

Home
Download
Documentation
Mailing Lists
Wiki
Bugs
Credits
Go to the first, previous, next, last section, table of contents.

1.30 Expansion operators

The following operations can be performed on portions of an expanded string:
```
${domain:<string>}
```
The string is interpreted as an RFC 822 address and the domain is extracted from it. If the string does not parse successfully, the result is empty.
```
${expand:<string>}
```
The `expand' operator causes a string to be expanded for a second time. For example,
```
${expand:${lookup{$domain}dbm{/some/file}{$value}}}
```
first looks up a string in a file while expanding the operand for `expand', and then re-expands what it has found.
```
${hash_<n>_<m>:<string>}
```
The two items <n> and <m> are numbers. If <n> is greater than or equal to the length of the string, the operator returns the string. Otherwise it computes a new string of length <n> by applying a hashing function to the string. The new string consists of characters t Exim Filter Specification - Expansion variables Go to the first, previous, next, last section, table of contents.

1.32 Expansion variables

This list of expansion variable substitutions contains those that are likely to be of use in filter files. Others that are not relevant at filtering time, or are of interest only to the system administrator, are omitted.

`$0', `$1', etc: When a `matches' expansion condition succeeds, these variables contain the captured substrings identified by the regular expression during subsequent processing of the success string of the containing "if" expansion item. They may also be set externally by some other matching process which precedes the expansion of the string. For example, the commands available in Exim filter files include an "if" command with its own regular expression matching condition.

`$domain': When an address is being directed, routed, or delivered on its own, this variable contains the domain. In particular, it is set during user filtering, but not during system filtering, since a message may have many recipients and the system filter is called just once.

`$home': This is set to the user's home directory when user filtering is configured in the normal way. When running a filter test via the `-bf' option, `$home' is set to the value of the environment variable HOME.

`$local_part': When an address is being directed, routed, or delivered on its own, this variable contains the local part. If a local part prefix or suffix has been recognized, it is not included in the value.

`$local_part_prefix': When an address is being directed or delivered locally, and a specific prefix for the local part was recognized, it is available in this variable. Otherwise it is empty.

`$local_part_suffix': When an address is being directed or delivered locally, and a specific suffix for the local part was recognized, it is available in this variable. Otherwise it is empty.

`$key': When a domain list is being searched, this variable contains the value of the key, so that it can be inserted into strings for query-style lookups. See chapter "" for details. In other circumstances this variable is empty.

`$message_body': This variable c Exim Overview
The Exim Mail Transport Agent

Table of Contents
- Exim Overview
Exim Overview

Date: 10 July 1997

Exim is a mail transport agent (MTA) developed at the University of Cambridge for use on Unix systems connected to the Internet. It is freely available under the terms of the GNU General Public Licence. In style it is similar to Smail 3, but its facilities are more extensive, and in particular it has some defences against mail bombs and unsolicited junk mail, in the form of options for refusing messages from particular hosts, networks, or senders.

Exim is in production use on a number of sites that move tens of thousands of messages per day. This document contains an overview description of the way Exim works, with a certain amount of omission and simplification to keep it fairly short. Please address any enquiries about Exim to Philip Hazel:
```
Email:  <ph10@cus.cam.ac.uk>
Phone:  +44 1223 334714
Fax:    +44 1223 334679

University of Cambridge
Computer Laboratory
Pembroke Street
Cambridge CB2 3QG
United Kingdom
```
This document is copyright (c) University of Cambridge 1997, but copying permission is granted to all. -------------------------------------------------------------------------

"If I have seen further it is by standing on the shoulders of giants."
(Isaac Newton)

Background

Exim owes a great deal to Smail 3 and its author, Ron Karr. Without the experience of running and working on the Smail 3 code, I could never have contemplated starting to write a new mailer. Many of the ideas and configuration interfaces are taken from Smail 3, though the actual code of Exim is entirely new.

My intention was to write a mailer that had more functionality than Smail 3, but which retained the simple lightweight approach, as this seemed to me to be all that was needed for systems directly connected to the Internet, where most messages are delivered almost immediately.

Availability

The current distribution of Exim is available from
```
$st{ftp://ftp.cus.cam.ac.uk/pub/software/programs/exim/exim-$si{n.nn}.tar.gz}
```
where n.nn is the version number. The distribution contains an ASCII copy of the documentation; other formats are available from
```
$st{ftp://ftp.cus.cam.ac.uk/pub/software/programs/exim/exim-postscript-$si{n.nn}.tar.gz}
$st{ftp://ftp.cus.cam.ac.uk/pub/software/programs/exim/exim-texinfo-$si{n.nn}.tar.gz}
```
The following operating systems are currently supported: AIX, BSDI, DGUX, FreeBSD, HI-OSF (Hitachi), HP-UX, IRIX, Linux, MIPS RISCOS, NetBSD, OpenBSD, DEC OSF1 (aka Digital UNIX), SCO, SCO SVR4.2 (aka UNIX-SV), SunOS4, SunOS5, Ultrix, and Unixware.

Limitations

For the benefit of those reading this overview to see whether Exim is of interest to them, its limitations are listed first.
- Exim is written in ANSI C. This should not be much of a limitation these days. However, to help with systems that lack a true ANSI C library, Exim avoids making any use of the value returned by the `sprintf()' function, which is one of the main incompatibilities. It has its own version of `strerror()' for use with SunOS4 and any other system that lacks this function, and a macro can be defined to turn `memmove()' into `bcopy()' if necessary.
- Exim uses file names that are longer than 14 characters.
- Exim is intended for use as an Internet mailer, and therefore handles addresses in RFC 822 domain format only. It cannot handle `bang paths', though simple two-component bang paths can be converted by a straightforward rewriting configuration.
- Exim insists that every address it handles has a domain attached. For incoming local messages, domainless addresses are automatically qualified with a configured domain value. Configuration options specify from which remote systems unqualified addresses are acceptable.
- The only external transport currently implemented is an SMTP transport over a TCP/IP network (using sockets), suitable for machines on the Internet. However, a pipe transport is available, and there are facilities for writing messages to files in `batched SMTP' format; both of these can be used to send messages to some other transport mechanism such as UUCP. Batched SMTP input is also catered for.
Main features

Exim follows the same general approach of decentralized control that Smail 3 does. There is no central process doing overall management of mail delivery. However, unlike Smail, the independent delivery processes share data in the form of `hints', which makes delivery more efficient in some cases. The hints are kept in a number of DBM files. If any of these files are lost, the only effect is to change the pattern of delivery attempts and retries.

Here is a summary of Exim's main features. More details are given in the sections which follow.
- Many configuration options can be given as expansion strings, and as these can include file lookups, much of Exim's operation can be made table-driven if desired. For example, it is possible to do local delivery on a machine on which the users do not have accounts.
- Regular expressions are available in a number of configuration parameters.
- Domain lists can include file lookups, making it possible to support a large number of local domains.
- Exim has flexible retry algorithms, applicable to mail routing as well as to delivery.
- Exim contains header and envelope rewriting facilities.
- Unqualified addresses are accepted only from specified hosts or networks.
- Exim can perform multiple deliveries down the same SMTP channel after deliveries to a host have been delayed.
- Exim can be configured to do local deliveries immediately but to leave remote deliveries until the message is picked up by a queue-runner process. This increases the likelihood of multiple messages being sent down a single SMTP connection.
- When copies of a message have to be delivered to more than one remote host, up to a configured maximum number of remote deliveries can be done in parallel.
- Exim supports optional checking of incoming return path (sender) and receiver addresses as they are received by SMTP.
- SMTP calls from specific hosts and networks, optionally from specific idents, can be locked out, and incoming SMTP messages from specific senders can also be locked out.
- It is possible to control which hosts may use the Exim host as a relay for onward transmission of mail; the control can be made to depend on the address domain.
- Messages on the queue can be `frozen' and `thawed' by the administrator.
- The maximum size of message can be specified.
- Exim can handle a number of independent local domains on the same machine; each domain can have its own alias files, etc. These are commonly called virtual domains.
- Exim stats a user's home directory before looking for a `.forward' file, in order to detect the case of a missing NFS mount.
- Exim contains an optional built-in mail filtering facility. This enables users to set up their own mail filtering in a straightfoward manner without the need to run an external program. There can also be a system filter file that applies to all messages.
- There is support for multiple user mailboxes controlled by prefixes or suffixes on the user name, either via the filter mechanism or through multiple `.forward' files.
- Periodic warnings are automatically sent to messages' senders when delivery is delayed -- the time between warnings is configurable.
- A queue run can be manually started to deliver just a particular portion of the queue, or those messages with a recipient whose address contains a given string.
- Exim can be configured to run as root only when needed; in particular, it need not run as root when receiving incoming messages or when sending out messages over SMTP. It always uses `setuid()' when doing local deliveries.
- I have tried to make the wording of delivery failure messages clearer and simpler, for the benefit of those less-experienced people who are now using email.
- Exim contains support for IPv6.
- The Exim Monitor is an optional extra; it displays information about Exim's processing in an X window, and an administrator can perform a number of control actions from the window interface.
Performance

Although I did not specifically set out to write a high-performance MTA, Exim does seem to be fairly efficient. The busiest site I know of is an mailing list exploder that sometimes handles over 100,000 deliveries a day on a big Linux box, the record being 177,000 deliveries (791MB in total). Up to 13,000 deliveries in an hour have been reported.

Interface

Like many MTAs, Exim has adopted the Sendmail interface so that it can be a straight replacement for `/usr/lib/sendmail'. All the relevant Sendmail options are implemented. There are also some additional options that are compatible with Smail 3, and some further options that are new to Exim.

The runtime configuration interface is a single file which is divided into a number of sections. The entries in this file consist of keywords and values, in the style of Smail 3 configuration files.

Control of messages on the queue can be done via certain privileged command line options. There is also an optional monitor program called `eximon', which displays current information in an X window and contains interfaces to the command line options.

Method of operation

When Exim receives a message, it writes two files in its spool directory. The first contains the envelope information, the current status of the message, and the headers, while the second contains the body of the message. The status of the message includes a complete list of recipients and a list of those that have already received the message. The header file gets updated during the course of delivery if necessary.

A message remains in the spool directory until it is completely delivered to its recipients or to an error address, or until it is deleted by an administrator or by the user who originally created it. In cases when delivery cannot proceed -- for example, when a message can neither be delivered to its recipients nor returned to its sender, the message is marked `frozen' on the spool, and no more deliveries are attempted. The administrator can thaw such messages when the problem has been corrected, and can also freeze individual messages by hand if necessary.

As delivery proceeds, Exim writes timestamped information about each address to a per-message log file; this includes any delivery error messages. This log is solely for the benefit of the administrator. All the information Exim itself needs for delivery is kept in the header spool file. The message log file is deleted with the spool files. If a message is delayed for more than a configured time, a warning message is sent to the sender. This is repeated whenever the same time elapses again without delivery being complete.

The main delivery processing elements of Exim are called directors, routers, and transports. Code for a number of these is provided, and compile-time options specify which ones are actually included in the binary. Directors handle addresses that include one of the local domains, routers handle remote addresses, and transports do actual deliveries.

When a message is to be delivered, the sequence of events is roughly as follows:
- If there is a system filter file, it is obeyed. This can check on the contents of the message and its headers, and cause delivery to be abandoned or directed to alternative or additional addresses.
- Each address is parsed and a check is made to see if it is local or not, by comparing the domain with the list of local domains, which can be wildcarded, or even held in a file if there are a large number of them.
- If an address is local, it is passed to each configured director in turn until one is able to handle it. If none can, the address is failed. Directors can be targeted at particular local domains, so several local domains can be processed independently of each other.
- A director that accepts an address may set up a local or a remote transport for it, or it may generate one or more new addresses (typically from alias or forward files). New addresses are fed back into this process from the top, but in order to avoid loops, a director will ignore any address which has an identically-named ancestor that was processed by itself.
- If an address is not local, it is passed to each router in turn until one is able to handle it. If none can, the address is failed.
- A router that accepts an address may set up a transport for it, or may pass an altered address to subsequent routers, or it may discover that the address is a local address after all. This typically happens when an partial domain name is used and (for example) the DNS lookup is configured to try to extend such names. In this case, the address is passed back to the directors.
- Routers normally set up remote transports for messages that are to be delivered to other machines. However, a router can pass a message to a local transport, and by this means messages for remote hosts can be routed to other transport mechanisms such as UUCP.
- When all the directing and routing is done, addresses that have been successfully handled are passed to their assigned transports. Local transports handle only one address at a time, but remote ones can handle more than one. Each local transport runs in a separate process under a non-privileged uid.
- If there were any errors, a message is returned to an appropriate address (the sender in the common case).
- If one or more addresses suffered a temporary failure, the message is left on the queue, to be tried again later. Otherwise the spool files and message log are deleted.
Mail filtering

Exim can be configured to allow users to set up filter files as an alternative to the traditional `.forward' files. A filter file can test various characteristics of a message, including the contents of the headers and the start of the body, and direct delivery to specified addresses, files, or pipes according to what it finds. The system-wide filter file uses the same control syntax.

Directors

The existing directors are listed below. I use the RFC 822 term local-part to mean that portion of an address that comes before the @ character.
- `aliasfile': This director handles local-part expansion via a traditional alias file. The name of the file is obtained by string expansion, and may therefore depend on the local-part or the domain. Generated pipe and file addresses can be (independently) locked out. The `aliasfile' director can also be used to test a list of local parts and direct any messages for them to a specific transport. In this case the data associated with the local part in the file is not used for address expansion, but is available for other purposes. For example, files containing records of the form
```
  foo: uid=1234 gid=5678 mailbox=/home_1/foo/inbox
```
  could be used on a system that did local deliveries without consulting its `passwd' file. The `aliasfile' director could use the file to verify that the local part was valid, and then the `appendfile' transport could use it to get a uid, gid, and mailbox for the delivery. The `aliasfile' director can also be configured to include the domain in the key that is looked up, making it possible to hold aliases for several different local domains in the same file.
- `forwardfile': This director handles local-part expansion via a traditional forward file or, if so configured, by a user's filter file. The name of the file is obtained by string expansion, and may therefore depend on the local-part or the domain, though if it is not an absolute path it is automatically assumed to be in the home directory of the user whose login name is the local-part. Mailing lists can be handled by file names of the form
```
  /some/list/directory/${local_part}
```
  and it is possible to specify an error address for each list that depends on the list name. Generated pipe and file addresses can be (independently) locked out.
- `localuser': This director matches the local-part of an address to a user of the machine. It can also be configured to do a pattern match on the user's home directory name. This makes it possible to partition the set of local users according to their home directories.
- `smartuser': This director matches any local-part. It can be used to pass messages for unknown users to a script that generates a helpful error message, or it can be used to send such messages to another host, optionally changing the envelope address in the process.
The configuration file determines which directors are actually used, and in which order. It is possible to use the same director more than once, with different options.

The addresses a director handles can be constrained in the following ways:
- A specific set of local domains may be specified, in which case the director is called only for addresses that contain one of those domains.
- A specific set of local parts may be specified, in which case the director is called only for addresses that contain one of those local parts. This could be used, for example, to handle `postmaster' independently of the particular local domain.
- A specific set of sender addresses may be specified. This can be used to implement closed mailing lists.
- A director may be configured to handle local-parts that start with a certain prefix and/or end with a certain suffix. For example, a director can be set up to handle local-parts of the form `xxxx-request' only.
- A flag controls whether a director is called when an address is being verified, as opposed to being directed for delivery.
In addition, certain files can be required to exist or not exist for a given director to be run.

Routers

The existing routers are:
- `domainlist:' This router searches a list of domains for the one it is trying to route. The list may either be a string in the configuration file, possibly including wild cards or regular expressions, or it may be in a file, or both may be provided. In the case of a file, keys of the form `*.foo.bar.com' can be used for simple wildcarding. If the domain is found, its entry can either specify a single replacement domain name that is passed on to subsequent routers, or it can specify a list of domain names that are looked up by this router. The lookup can be done by the `gethostbyname' function, or by DNS lookup, and in the latter case it is configurable whether MX or A records or both are used. As well as providing explicit routing for certain domains, the domainlist router can be used to set up gateways for partial domains (e.g. for *.uucp) and it can also be used as a `smarthost' router by using the all-inclusive wild card.
- `lookuphost:' This router looks up domain names either by calling the `gethostbyname' function, or by using the DNS. In the latter case, it can be configured to use the DNS resolver options for qualifying single-component names and for searching parent domains. It is also possible to specify explicit text strings for widening domains that are not found initially. It is possible to insist on the presence of MX records for certain sets of domains. A configuration option controls whether the message's headers are rewritten when a domain name is changed.
- `queryprogram:' This router passes the address to a script that runs in a separate process under an unprivileged uid and gid. The script returns a line of text specifying whether it matched the domain or not. If it did match, it may specify a transport name, or it may specify that the transport specified for the router is used. The script may also send back a new domain name to replace the current one, and specify a method of looking this name up (gethostbyname, DNS, or pass to next router).
The configuration file determines which routers are actually used, and in which order. It is possible to use the same router more than once, with different options.

Like directors, routers can be constrained to handle only certain domains or certain local parts (though I haven't seen a good use for that yet), or messages from certain senders. If a router times out, either the delivery can be deferred, or the address can be passed on to the next router.

A flag controls whether a router is called when an address is being verified, as opposed to being routed for delivery.

Transports

Local and remote transports are handled differently. A local transport is always run in a separate process with an appropriate real uid and gid. Their values can be specified in the transport's configuration, or passed over from the director that handled the address. Remote transports run under exim's own uid. The existing transports are:
- `appendfile:' This local transport appends the message to a file whose name is specified as a string containing variable expansions. The current local-part can be inserted via the expansion mechanism, and file names such as
```
/home/${local_part}/inbox
/var/mail/${local_part}
```
  are typical examples. However, it is possible to look up each individual user's inbox name in a file, should that be required. Exclusive access to the file is ensured by using the traditional mailbox locking strategy of creating a lock file. The lock creation process uses a `hitching post' algorithm (similar to that used by Pine) which is robust when the mailbox file is NFS-mounted. The file is also locked using the `lockf' function, but either locking mechanism can be turned off if necessary. The `appendfile' transport can also be configured to deliver each message into a separate file in a given directory, optionally in `maildir' format. Options on this transport allow for the insertion of a prefix line (e.g. `From xxx...') and suffix line, special processing of message lines starting with `From', and the addition of `Return-path', `Delivery-date', and `Envelope-to' headers. If the mailbox file is not a regular file, or does not have the correct owner, group, or permissions, no delivery takes place; the address is deferred and the postmaster is informed, except that, if the file's permissions are greater than those required, Exim reduces the permissions and carries on. There are additional checks to reduce the possibility of security exposures caused by race conditions.
- `pipe': This local transport passes the message via a pipe to a specified command (program or script) which is run in a separate process under a given uid and gid. Various parameters of the message are passed as environment variables, and there are the same options as for `appendfile' for controlling the form of the message. The returned status of the command may be used to determine success or failure, or it can be ignored. A configuration option specifies whether any standard output generated by the command is to be returned to the sender (either always, or only if the command fails). If this is set unconditionally and output is actually generated, the delivery is deemed to have failed, whatever the returned status of the command. The maximum amount of output generated by the command can be controlled, and a timeout may be set for it.
- `smtp': This remote transport delivers a message using SMTP over TCP/IP. All addresses in the message that route to the same set of hosts, and have the same errors address (return path), are normally sent in a single transaction. An explicit list of hosts can be set for the transport, or a host list may be attached to an address by one of the routers. If all the hosts are temporarily unable to accept the message, it is delivered to one of a list of fallback hosts, if configured.
Exim logs

Exim write four different log files:
- The main log records the arrival of each message and the result of each delivery attempt in a single line in each case. The format is as compact as possible, in an attempt to keep down the size of log files. A number of other events are also recorded on the main log.
- The reject log records information from messages that are rejected because their return paths are invalid (a configurable option), or the sender or sending host appears in a blocking list. The headers are written to this log if they have been read, following a copy of the one-line message that is also written to the main log.
- The panic log is written when Exim suffers a disaster and has to bomb out.
- On systems that support signal handlers that restart a system call on exit, Exim reacts to a USR1 signal by writing a line describing its current activity to the process log. This makes it possible to find out what each exim process on a machine is currently doing.
A utility script for renaming and compressing the main and reject logs each night is provided. There are also scripts for extracting statistics from log files and for searching log files for the entries for messages that match a given pattern. For example, one can pull out all entries relating to messages for a given local part.

Exim databases

Exim maintains a number of databases in DBM files to help it perform efficient mail delivery. In effect, the files contain hints, and if they are lost it is not a disaster -- Exim's performance just suffers a bit. The three databases currently used are:
- `retry': This contains information about each failing remote host and temporary failing local delivery -- when the first failure was detected, when the delivery (or directing or routing) was last tried, and when it should next be tried. More details about retry algorithms are given below.
- `wait-smtp': This contains information about messages that are waiting for particular hosts after an SMTP delivery failure (see the next section).
- `reject': This contains information about SMTP message rejections (see below).
There is a utility program that lists the contents of one of these databases, and another that allows manual modifications to be applied in some cases. Database records are timestamped, and there is a utility that removes records that are older than a given period, and also cleans up `wait-smtp' records containing references to messages that no longer exist. Running this daily or weekly should be sufficient to keep the files reasonably tidy.

Exim can use any one of the following DBM libraries: `ndbm', `gdbm', `DB 1.85', or `DB 2.x'.

SMTP batching

When an SMTP delivery attempt fails, causing the message to be deferred till later, Exim updates a DBM database that contains records keyed by host name plus IP address. Each record holds a list of messages that are waiting for that host and address.

When an SMTP delivery succeeds, Exim consults the database to see if there are any other messages waiting for the same host and address. If it finds any, it creates a new Exim process and passes it the open SMTP channel and a message identification. The new process then delivers the waiting message down the existing channel and may in turn cause the creation of yet another process. Any other waiting addresses in the message are skipped. The maximum number of messages sent down one connection is configurable.

This scheme achieves some SMTP efficiency when a number of messages have been queued up for a given host, without the overhead of a heavyweight queueing apparatus.

Retries

When a message cannot immediately be directed, routed, or deli Exim Specification - 1 Introduction Go to the first, previous, next, last section, table of contents.

1 Introduction

"If I have seen further it is by standing on the shoulders of giants."
(Isaac Newton)

Exim is a mail transfer agent (MTA) for Unix systems connected to the Internet. Configuration files currently exist for the following operating systems: AIX, BSDI, DGUX, FreeBSD, HI-OSF (Hitachi), HP-UX, IRIX, Linux, MIPS RISCOS, NetBSD, OpenBSD, DEC's OSF1 (aka Digital UNIX), SCO, SCO SVR4.2 (aka UNIX-SV), SunOS4, SunOS5, Ultrix, and Unixware. However, code is not available for determining system load averages for AIX or Ultrix.

The terms and conditions for the use and distribution of Exim are contained in the file `NOTICE'. Exim is distributed under the terms of the GNU General Public Licence, a copy of which may be found in the file `LICENCE'.

Exim owes a great deal to Smail 3 and its author, Ron Karr. Without the experience of running and working on the Smail 3 code, I could never have contemplated starting to write a new mailer. Many of the ideas and user interfaces are taken from Smail 3, though the actual code of Exim is entirely new.

I am indebted to my colleague Piete Brooks for suggesting and implementing the scheme for building Exim for multiple architectures and operating systems, for porting Exim to several different versions of Unix, and for numerous suggestions. Many other people, both in Cambridge and around the world, have contributed to the development and the testing of Exim, and to porting it to various operating systems. I am grateful to them all.

This edition of the Exim specification applies to version 1.90 of Exim. Substantive changes from the 1.80 edition are marked by bars in the right margin, except in the Texinfo version of the documentation, because Texinfo doesn't support change bars. Minor corrections and rewordings are not so marked.

As the program is still developing, there may be features in later versions of the program that have not yet made it into this document, which is updated only when there is a reasonable batch of changes to make. However, all changes are noted briefly in the distributed file called `doc/ChangeLog', and specifications of new features that are not in this manual are placed in `doc/NewStuff'.

1.1 Mailing l

The Exim Mail Transport Agent

Table of Contents