MTR network troubleshooting guide

If you're not comfortable with the command line, use a ready-made tool

# If you're not comfortable with the command line, use a ready-made tool

BestTrace route tracing tool
Web-based traceroute

# MTR command explained

MTR is a powerful network diagnostic tool. It combines the features of traceroute and ping and collects more information — such as connection state and availability — which is very useful when troubleshooting network problems.

mtr (My traceroute) comes pre-installed on almost every Linux distribution. It merges the functions of ping and traceroute, making it more powerful. By default mtr sends ICMP packets to probe the link. You can also use the -u option to probe with UDP packets. Unlike traceroute, which only runs one trace, mtr keeps probing the nodes along the link continuously and gives corresponding statistics. This lets mtr avoid the impact of node fluctuations on the result, so its results are more accurate — we recommend using it first.

Installing MTR

On Linux you can install it directly with yum (CentOS example):

[root@host ~]# yum provides mtr      # check which package provides the mtr command
Loaded plugins: fastestmirror
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Determining fastest mirrors

 * remi-php80: mirrors.tuna.tsinghua.edu.cn
 * remi-safe: mirrors.tuna.tsinghua.edu.cn
   2:mtr-0.85-7.el7.x86_64 : A network diagnostic tool
   Repo        : base

[root@host ~]# yum install mtr -y     # install mtr

On Windows, we recommend installing besttrace, which includes MTR functionality.

On macOS, we recommend installing via brew.

mtr usage

The most basic use of mtr is simple — just run: mtr <ip or domain>

Usage

# mtr -h
usage: mtr [-BfhvrwctglxspQomniuT46] [--help] [--version] [--report]
                [--report-wide] [--report-cycles=COUNT] [--curses] [--gtk]
                [--csv|-C] [--raw] [--xml] [--split] [--mpls] [--no-dns] [--show-ips]
                [--address interface] [--filename=FILE|-F]
                [--ipinfo=item_no|-y item_no]
                [--aslookup|-z]
                [--psize=bytes/-s bytes] [--order fields]
                [--report-wide|-w] [--inet] [--inet6] [--max-ttl=NUM] [--first-ttl=NUM]
                [--bitpattern=NUM] [--tos=NUM] [--udp] [--tcp] [--port=PORT] [--timeout=SECONDS]
                [--interval=SECONDS] HOSTNAME

Common optional parameters

--report: show output in report mode.
--split: list the result of each trace separately, rather than aggregating the whole result.
--psize: set the size of the ping packet.
--no-dns: do not do reverse DNS lookup on IP addresses.
--address: when the host has multiple IPs, set the source IP used to send packets.
-4: use IPv4 only.
-6: use IPv6 only.

You can also press keys like the following while mtr is running to switch modes quickly:

? or h: show the help menu.
d: toggle display mode.
n: enable/disable DNS resolution.
u: switch between ICMP and UDP probing.

MTR parameters in detail

 -F, --filename FILE        read hostname(s) from a file
 -4                         use IPv4 only
 -6                         use IPv6 only
 -u, --udp                  use UDP instead of ICMP echo
 -T, --tcp                  use TCP instead of ICMP echo
 -a, --address ADDRESS      bind the outgoing socket to ADDRESS
 -f, --first-ttl NUMBER     set what TTL to start
 -m, --max-ttl NUMBER       maximum number of hops
 -U, --max-unknown NUMBER   maximum unknown host
 -P, --port PORT            target port number for TCP, SCTP, or UDP
 -L, --localport LOCALPORT  source port number for UDP
 -s, --psize PACKETSIZE     set the packet size used for probing
 -B, --bitpattern NUMBER    set bit pattern to use in payload
 -i, --interval SECONDS     ICMP echo request interval
 -G, --gracetime SECONDS    number of seconds to wait for responses
 -Q, --tos NUMBER           type of service field in IP header
 -e, --mpls                 display information from ICMP extensions
 -Z, --timeout SECONDS      seconds to keep probe sockets open
 -r, --report               output using report mode
 -w, --report-wide          output wide report
 -c, --report-cycles COUNT  set the number of pings sent
 -j, --json                 output json
 -x, --xml                  output xml
 -C, --csv                  output comma separated values
 -l, --raw                  output raw format
 -p, --split                split output
 -t, --curses               use curses terminal interface
     --displaymode MODE     select initial display mode
 -n, --no-dns               do not resove host names
 -b, --show-ips             show IP numbers and host names
 -o, --order FIELDS         select output fields
 -y, --ipinfo NUMBER        select IP information in output
 -z, --aslookup             display AS number
 -h, --help                 display this help and exit
 -v, --version              output version information and exit

Example

[[email protected] ~]# mtr -rn www.baidu.com
Start: Mon Jul 26 16:33:13 2021
HOST: iZbp1377m2iy8ro0dbnt1kZ     Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 10.12.208.110              0.0%    10    2.7   2.6   2.5   2.9   0.0
  2.|-- 10.12.208.73              10.0%    10    5.0   4.9   4.1   8.2   1.2
  3.|-- 10.255.101.109             0.0%    10    2.5   3.0   2.4   7.8   1.6
  4.|-- 103.41.142.162             0.0%    10    3.5   3.7   3.4   5.4   0.5
  5.|-- 10.102.46.61               0.0%    10    2.9   3.0   2.9   3.1   0.0
  6.|-- 115.238.21.14              0.0%    10    3.1   3.0   3.0   3.1   0.0
  7.|-- 220.191.199.73             0.0%    10    8.1   6.7   6.5   8.1   0.5
  8.|-- 202.97.33.145             30.0%    10   13.9  14.5  13.9  15.9   0.6
  9.|-- 58.213.95.98              80.0%    10   14.8  14.8  14.8  14.9   0.0
 10.|-- 58.213.95.130             90.0%    10   14.6  14.6  14.6  14.6   0.0
 11.|-- 58.213.96.78               0.0%    10   13.0  12.9  12.8  13.0   0.0
 12.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
 16.|-- 180.101.49.12              0.0%    10   14.0  14.0  14.0  14.0   0.0

Result fields

With the default configuration, the columns mean:

Column 1 (Host): node IP and domain. Press n to toggle the display.
Column 2 (Loss%): packet loss rate at the node.
Column 3 (Snt): packets sent per second. Default is 10; can be set with -c.
Column 4 (Last): the latency of the most recent probe.
Columns 5/6/7 (Avg, Best, Worst): the average, minimum and maximum probe latency.
Column 8 (StDev): standard deviation; the larger it is, the more unstable that node.

Judging Loss% (packet loss)

If any node's Loss% is non-zero, that hop may have a problem. There are usually two causes of loss at a node:

The carrier limits the node's ICMP send rate for security or performance reasons, causing loss.
The node really has an anomaly, causing loss.

Combine the loss at the abnormal node and its subsequent nodes with the following to judge the cause:

If none of the subsequent nodes lose packets, the loss at the abnormal node is usually due to a carrier policy limit and can be ignored.
If subsequent nodes also lose packets, the abnormal node usually has a real network anomaly causing loss.
The two cases above can also happen together — a node may have both rate limiting and a network anomaly. In that case, if the abnormal node and the following nodes have continuous loss with different rates, the loss rates of the last few hops are usually authoritative.

About timeouts

Sometimes you may see (???) in the mtr output. This can be caused by some routers dropping ICMP and not replying, leading to a timeout, or by a problem on the return path.
A timeout is not necessarily an indication of packet loss. Packets can still reach their destination without obvious loss or delay. Timeouts may be due to routers dropping packets for QoS (quality of service) purposes, or to some issue on the return route.

Command examples

Use -r to show a report (the default is dynamic display):

mtr -r www.badu.com

Use -c to set the number of packets sent per second:

mtr -r -c 30 www.baidu.com

Use -s to set the ping packet size:

mtr -r -c 30 -s 1024 www.baidu.com