Nomad servers may need to be run on large machine instances. We suggesthaving between 4-8+ cores, 16-32 GB+ of memory, 40-80 GB+ of fast disk andsignificant network bandwidth. The core count and network recommendations are toensure high throughput as Nomad heavily relies on network communication and asthe Servers are managing all the nodes in the region and performing scheduling.The memory and disk requirements are due to the fact that Nomad stores all statein memory and will store two snapshots of this data onto disk, which causes high IO in busy clusters with lots of writes. Thus disk shouldbe at least 2 times the memory available to the server when deploying a highload cluster. When running on AWS prefer NVME or Provisioned IOPS SSD storage for data dir.
These recommendations are guidelines and operators should always monitor theresource usage of Nomad to determine if the machines are under or over-sized.
Nomad clients support reserving resources on the node that should not beused by Nomad. This should be used to target a specific resource utilization pernode and to reserve resources for applications running outside of Nomad'ssupervision such as Consul and the operating system itself.
Please see the reservation configuration formore detail.
Network Topology
Nomad servers are expected to have sub 10 millisecond network latenciesbetween each other to ensure liveness and high throughput scheduling. Nomadservers can be spread across multiple datacenters if they have low latencyconnections between them to achieve high availability.
For example, on AWS every region comprises of multiple zones which have very lowlatency links between them, so every zone can be modeled as a Nomad datacenterand every Zone can have a single Nomad server which could be connected to form aquorum and a region.
Nomad servers uses Raft for state replication and Raft being highly consistentneeds a quorum of servers to function, therefore we recommend running an oddnumber of Nomad servers in a region. Usually running 3-5 servers in a region isrecommended. The cluster can withstand a failure of one server in a cluster ofthree servers and two failures in a cluster of five servers. Adding more serversto the quorum adds more time to replicate state and hence throughput decreasesso we don't recommend having more than seven servers in a region.
Nomad clients do not have the same latency requirements as servers since theyare not participating in Raft. Thus clients can have 100+ millisecond latency totheir servers. This allows having a set of Nomad servers that service clientsthat can be spread geographically over a continent or even the world in the caseof having a single "global" region and many datacenter.
Nomad clients make connections to servers on the RPC port and then maintain apersistent TCP connection. The server and client use this TCP connection fortwo-way communication. As a result, clients that are geographically distributedfrom the servers do not need to have publically routable IP addresses in orderto communicate with the servers (although the workloads running on the clientsmay need public IPs). All connections between Nomad servers and between clientsand servers must be secured with mTLS.
Nomad clients are typically not required to be reachable from each other unlessyour workloads need to communicate with each other. The optional ephemeral diskmigration field is one exception, and requires that clients can reach eachother on their HTTP ports.
Nomad requires 3 different ports to work properly on servers and 2 on clients,some on TCP, UDP, or both protocols. Below we document the requirements for eachport. If you use a firewall of any type, you must ensure that it is configured toallow the following traffic.
HTTP API (Default 4646). This is used by clients and servers to serve the HTTPAPI. TCP only.
RPC (Default 4647). This is used for internal RPC communication between clientagents and servers, and for inter-server traffic. TCP only.
Serf WAN (Default 4648). This is used by servers to gossip both over the LAN andWAN to other servers. It isn't required that Nomad clients can reach this address.TCP and UDP.
When tasks ask for dynamic ports, they are allocated out of the port rangebetween 20,000 and 32,000. This is well under the ephemeral port range suggestedby the IANA. If your operatingsystem's default ephemeral port range overlaps with Nomad's dynamic port range,you should tune the OS to avoid this overlap.
On Linux this can be checked and set as follows:
$ cat /proc/sys/net/ipv4/ip_local_port_range32768 60999$ echo "49152 65535" > /proc/sys/net/ipv4/ip_local_port_rangeBridge Networking and iptables
Nomad's task group networks and Consul Connect integration use bridge networking and iptables to send traffic between containers. The Linux kernel bridge module has three "tunables" that control whether traffic crossing the bridge are processed by iptables. Some operating systems (RedHat, CentOS, and Fedora in particular) configure these tunables to optimize for VM workloads where iptables rules might not be correctly configured for guest traffic.
These tunables can be set to allow iptables processing for the bridge network as follows:
$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptablesTo preserve these settings on startup of a client node, add a file including the following to /etc/sysctl.d/ or remove the file your Linux distribution puts in that directory.
net.bridge.bridge-nf-call-arptables = 1net.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1On Linux, Nomad uses cgroups to control access to resources like CPU andmemory. Nomad can support both cgroups v2 and the legacy cgroupsv1. When Nomad clients start, they will determine the available cgroupcontrollers and include the attribute os.cgroups.version in theirfingerprint.
Nomad can only use cgroups to control resources if all the required controllersare available. If one or more required cgroups are unavailable, Nomad willdisable resource controls that require cgroups entirely. You will most often seemissing controllers on platforms used outside of datacenters, such as RaspberryPi or similar hobbyist computers.
On cgroups v2, you can verify that you have all required controllers as follows:
$ cat /sys/fs/cgroup/cgroup.controllerscpuset cpu io memory pidsOn legacy cgroups v1, you can look for this same list of required controllers asdirectories under the directory /sys/fs/cgroup.
To enable missing cgroups, add the appropriate boot command line arguments. Forexample, to enable the cpuset cgroup, you'll need to add cgroup_cpuset=1cgroup_enable=cpuset. These arguments should be added wherever specified byyour bootloader.
Hardening Nomad
As noted in the Security Model guide, Nomad is not secure-by-default.
User Permissions
Nomad servers and Nomad clients have different requirements for permissions.
Nomad servers should be run with the lowest possible permissions. They needaccess to their own data directory and the ability to bind to their ports. Youshould create a nomad user with the minimal set of required privileges. If youare installing Nomad from the official Linux packages, the systemd unit fileruns Nomad as root. For your server nodes you should change this to aminimally privileged nomad user. See the production deployment guide fordetails.
Nomad clients must be run as root due to the OS isolation mechanisms thatrequire root privileges (see also Linux Capabilities below). The Nomadclient's data directory should be owned by root with filesystem permissionsset to 0700.
Linux Capabilities
On Linux, Nomad clients require privileged capabilities for isolatingtasks. Nomad clients require CAP_SYS_ADMIN for creating the tmpfs used forsecrets, bind-mounting task directories, mounting volumes, and running some taskdriver plugins. Nomad clients require CAP_NET_ADMIN for a variety of tasks toset up networking. You should run Nomad clients as root, but running as rootdoes not grant these required capabilities if Nomad is running in a usernamespace. Running Nomad clients inside a user namespace is unsupported. See thecapabilities(7) man page for details on Linux capabilities.
In order to run a task, Nomad clients perform privileged operations normallyreserved to the root user:
- Mounting tmpfs file systems for the task
/secretsdirectory. - Creating the network bridge for
bridgenetworking. - Allowing inbound and outbound network traffic to the workload (typically via
iptables). - Starting tasks as a specific
user. - Setting the owner of
templateoutputs.
On Linux this set of requirements expands to:
- Configuring resource isolation via cgroups.
- Configuring namespace isolation:
mount,user,pid,ipc, andnetworknamespaces.
Nomad task drivers that support bind-mounting volumes also need to run as rootto do so. This includes the built-in exec and java task drivers. Thebuilt-in task drivers run in the same process as the Nomad client, so thisrequires that the Nomad client agent is also running as root.
Rootless Nomad Clients
Although it's possible to run a Nomad client agent as a non-root user or asroot in a user namespace, to perform the privileged operations described aboveyou also need to grant the client agent CAP_SYS_ADMIN and CAP_NET_ADMINcapabilities. Note that these capabilities are nearly functionally equivalent torunning as root and that a process running with CAP_SYS_ADMIN can almostalways escalate itself to "true" (unnamespaced) root.
Some task drivers delegate many of their privileged operations to an externalprocess such as dockerd or podman. If you don't need bridge networking andare using these task drivers or custom task drivers, you may be able to runNomad client agents as a non-root user with the following additionalconfiguration:
- Delegated cgroups: to safely set cgroups as an unprivileged user requirescgroups v2.
- User namespaces: on some distros this may require setting sysctls like
kernel.unprivileged_userns_clone=1 - The task driver engine (ex.
dockerd,podman,containerd, etc) must beconfigured for rootless operation. This requires cgroups v2, user namespaces,and typically either a patched kernel or kernel module (ex.overlay.ko)allowing unprivileged overlay filesystem or a FUSE overlay filesystem.
This is not a supported or well-tested configuration. See GH-13669 for afurther discussion and to provide feedback on your experiences trying to runrootless Nomad clients.
Running systems as Docker containers has become a common practice. While it'spossible to run Nomad servers inside containers, Nomad clients requireextensive access to the underlying host machine, as described inRootless Nomad Clients. Docker containers introduce a non-trivialabstraction layer that makes it hard to properly configure clients and taskdrivers therefore running Nomad clients in Docker containers is notofficially supported.
The hashicorp/nomad Docker image is intended to be usedin automated pipelines for CLI operations, such asnomad job plan, nomad fmt, and others.
Note: The Nomad Docker image is not tested when running as an agent.
