[jds]

This is not a question but a post-mortem for posterity.
I hope it might serve as a guide to others experiencing similar problems.

* Yesterday, I discovered the ARP entry for my default gateway was
failing, becoming incomplete for several seconds at a time.

* Symptoms were: a tunnel terminated behind the dd-wrt router would
lose connection sporadically; the dd-wrt router would sporadically
return Host Unreachable ICMP errors to pings from LAN to Internet.

* Unicast/broadcast requests sent with arping would elicit replies
only once every 30 seconds or so. (It varied.)

* `ip -s neigh` indicated a delay had occurred well before the
default `net.ipv4.neigh.vlan2.base_reachable_time` (30) was
reached. (`vlan2` is WAN.)

* Increasing `...base_reachable_time` and `...gc_stale_time` to 60
and 120, respectively, was not sufficient to stop ARP failures.

* Setting `...base_reachable_time` and `...gc_stale_time` to 150 and
300, respectively, eliminated further ARP failures.

* No tunnel or packet drops have occurred since.


## Symptoms

### Host Unreachables

[1490129293.654161] From 192.0.2.92 icmp_seq=75 Destination Host Unreachable
[1490129293.654506] From 192.0.2.92 icmp_seq=76 Destination Host Unreachable
[1490129293.654514] From 192.0.2.92 icmp_seq=77 Destination Host Unreachable
[1490129296.712800] From 192.0.2.92 icmp_seq=78 Destination Host Unreachable
[1490129296.712815] From 192.0.2.92 icmp_seq=79 Destination Host Unreachable
[1490129296.712817] From 192.0.2.92 icmp_seq=80 Destination Host Unreachable
[1490129723.859703] From 192.0.2.92 icmp_seq=505 Destination Host Unreachable
[1490129723.859724] From 192.0.2.92 icmp_seq=506 Destination Host Unreachable
[1490129938.088795] From 192.0.2.92 icmp_seq=719 Destination Host Unreachable
[1490129938.088810] From 192.0.2.92 icmp_seq=720 Destination Host Unreachable
[1490129941.116116] From 192.0.2.92 icmp_seq=723 Destination Host Unreachable
[1490129944.124625] From 192.0.2.92 icmp_seq=726 Destination Host Unreachable
[1490129947.122741] From 192.0.2.92 icmp_seq=728 Destination Host Unreachable
[1490129947.123003] From 192.0.2.92 icmp_seq=729 Destination Host Unreachable
[1490129950.149668] From 192.0.2.92 icmp_seq=731 Destination Host Unreachable
[1490129950.149683] From 192.0.2.92 icmp_seq=732 Destination Host Unreachable
[1490129953.150300] From 192.0.2.92 icmp_seq=735 Destination Host Unreachable
[1490129953.150312] From 192.0.2.92 icmp_seq=736 Destination Host Unreachable

### ARP Failures

192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 17/17/17 REACHABLE
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/18/0 DELAY
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/19/1 DELAY
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/20/2 DELAY
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/21/4 DELAY
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/22/0 PROBE
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/23/1 PROBE
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/24/2 PROBE
192.0.2.1 dev vlan2 used 0/25/0 FAILED
192.0.2.1 dev vlan2 ref 1 used 0/26/1 INCOMPLETE
192.0.2.1 dev vlan2 ref 1 used 0/27/2 INCOMPLETE
192.0.2.1 dev vlan2 ref 1 used 0/28/0 INCOMPLETE
192.0.2.1 dev vlan2 ref 1 used 0/29/0 INCOMPLETE
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/0/0 REACHABLE

### Fix

I set `base_reachable_time` and `gc_stale_time` for the WAN interface (vlan2) in
Administration->Commands->Startup.

sysctl -w net.ipv4.neigh.vlan2.gc_stale_time=300
sysctl -w net.ipv4.neigh.vlan2.base_reachable_time=150

### ARP After Fix

192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 82/82/31 REACHABLE
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/83/0 DELAY
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/85/1 DELAY
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/86/2 DELAY
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/87/3 DELAY
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/88/4 DELAY
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/89/1 PROBE
192.0.2.1 dev vlan2 lladdr 00:01:5c:63:18:46 ref 1 used 0/0/0 REACHABLE