Path visualisation

Slow on the path? See whose network it is.

A traceroute lists the routers between two points, and each router is a hop. FishEye runs one to each target twice a minute by default. It shows every hop with the network that runs it and where it is, marks the hop where the delay starts, and keeps the history so you can see what the route looked like before.

Path visualisation is included from plan T2.

The first five hops are fine. The delay starts at the handover.

FishEye puts the answer above the table. In this recording of a demo network, the first five hops answer in 4 ms or less. Then 218.6 ms appear in one step, between hop 5 on DEMO-CARRIER and hop 6 on DEMO-TRANSIT. That is the network to call.

  • The callout names both ends of the jump. A rise belongs to the link between two routers, not to the far one.
  • It says how much of the end-to-end delay the jump is. Here, 95%.
  • A hop is marked as the bottleneck only if the delay stays up for every hop after it.
The Path Visualization hop table for a demo target. A warning bar reads: Latency rises by 218.6 ms between hop 5 and hop 6 and stays up to the last reply: 95% of the end-to-end latency. Hop 6 carries a Bottleneck marker and is on AS64501 DEMO-TRANSIT in Cape Town, after five hops at 4 ms or less.
The callout and the bottleneck marker say where the 218.6 ms appear and whose hop it is. The real console on a demo network: hosts and networks are made up.

Every hop has a network and a place.

Each hop shows the network that runs it, then where it is. A network is named by its ASN, the number it is known by on the internet, beside the operator. Hop 3 here is AS64500, DEMO-CARRIER, in Johannesburg.

  • The network and operator come from Team Cymru’s public lookup. Places are measured by RIPE IPmap where it can, and from a local database where it cannot.
  • A place is kept only if the hop’s own round trip makes it possible. Light in fibre covers about 200 km each millisecond, so a hop that answers in 2 ms cannot be more than about 200 km away.
  • Weaker places are tagged, for example unverified, or no fixed location for an address announced from many sites.
  • Private addresses are not looked up with outside services. They get a label instead: local network, carrier access, transit or target, read from the address range.
Eight hops of a trace. Hops 1 and 2 are a local router and carrier access, labelled but with no network. Hops 3 to 5 are in Johannesburg on AS64500 DEMO-CARRIER. Hops 6 and 7 are in Cape Town on AS64501 DEMO-TRANSIT. Hop 8 is the target.
Network and place on every hop that has a public address. The real console on a demo network.

Scroll back to the route before.

The rail beside the map is the Time Machine. The top is now. Scroll down and the map, the tiles and the hop table show that moment. Live refresh pauses until you go back to live.

  • Every run is a bar on the rail. The longer the bar, the higher the latency.
  • Look back over 1 hour, 6 hours, 24 hours, 3, 7 or 30 days, or a range you pick. Or type the exact time you want.
  • A flag on the rail marks each route change. The change log under the map lists them.
  • You can switch between the master and each probe node, to see the same target from another site.
The Path Visualization Time Machine scrolled back in time. The reading line on the rail says 13:22:10, 17.3 ms, 8 hops. The map shows the target core-cpt-02 on its earlier route with every hop green. A rail of green bars on the right runs back through the earlier hours. A Download TXT button and a Use current snapshot link appear above the map, and the change log lists one route change.
Scrolled back to before the change: the same target on its earlier route, 17.3 ms end to end. Live refresh is paused and Download TXT appears. The real console on a demo network.

A slow router is not a slow path.

A traceroute uses ICMP, the message type a ping uses. Many routers treat those replies as low priority, so a hop can look slow while the traffic going through it is not delayed at all.

FishEye compares each hop with the hops after it. If later hops answer faster, the hop is tagged slow ICMP reply and is not called the bottleneck.

A hop table for a demo target with six hops. Hop 4 answers in 37.4 ms and carries a slow ICMP reply tag. Hop 5 answers in 2.5 ms and the target in 3.1 ms, so the path itself is fast.
Hop 4 answers in 37.4 ms, the target in 3.1 ms. FishEye tags the hop and does not blame it. The real console on a demo network.

Take it to your provider.

A carrier ticket needs three things: where the delay starts, whose network that is, and what changed. FishEye shows all three on one page.

  • The hop and its network: hop 6, on AS64501 DEMO-TRANSIT.
  • The route change log names the first hop that differs, the old and the new address, when the new route was first seen and when it was confirmed.
  • Copy a trace as plain text, or download the trace of a past moment as a text file, and paste it into the ticket.

There is no share link or path report to export yet: you paste the text and the before-and-after addresses into the ticket.

The Path Visualization change log with one entry: Route change, core-cpt-02 from the master, hop 6 moved to a different router. First seen on the new route, and confirmed a minute later.
The log names the first hop that moved, hop 6, and its old and new address, with the times the new route was first seen and confirmed. Hosts and networks are made up.

A recorded investigation

The real FishEye console, recorded on a demo network. A target slows down, and from step 4 Path Visualization finds the hop where the latency starts. Host names and addresses are made up, and differ from the pictures above.

How FishEye judges a route change.

A route that flips back and forth is not news. FishEye records a change only when it is sure of it.

A change is recorded when
Three runs in a row take the new route, and that source has not used it in the last 7 days. On one day of our own network’s traces, the simpler rule of three runs alone found 582 changes, most of them returns to a known route. This rule found 11.
How often it traces
From the master, twice a minute by default, and you can set 1 to 6 a minute per target. From each probe node, every few minutes. Only targets set to traceroute or full are traced.
How long it keeps it
Per-hop detail for 90 days by default, which you can change. Route changes and hourly summaries for about 400 days.
How it traces
With ICMP echo, the same message a ping uses, because some networks drop the UDP probes a traceroute can also use. If the traceroute program lacks permission to send ICMP echo, it falls back to UDP.
Alerts
A path-change alert is optional and off until you add a trigger. It opens once for each episode.
What it looks up outside
Only public hop addresses, for their network and place. Private addresses are not sent to outside services.

A known limit. On a path where nearly every run takes a different route, FishEye cannot yet tell a real change from the normal spread, so it can miss changes there. We are looking at a better way to trace such paths.

Show us the path that keeps going wrong.

Tell us which carrier you chase most, and we will show you what FishEye finds on its path.