George's Weblog

Proxy arp is a nice feature to have when you're making changes in the network and need things to keep working along the way. It's real easy to use on a Cisco or Juniper router but there are a few caveats when using the feature on pfSense.

To understand what proxy arp does, think of the following situation:

You have a LAN with a network of 10.10.10.0/24.

Elsewhere on the network you have a different network segment, say 10.20.20.0/24. On that segment is a host with the address 10.20.20.100.

You want traffic that used to go to 10.10.10.100 to go to the host at 10.20.20.100 instead. Assume you don't have control over where the traffic is coming from and your only choice is to take the traffic 'as-is' on your network.

The first step in the process is to multihome the server at 10.20.20.100. You want to bring up the IP address 10.10.10.100 on it with a /32 mask (just the one single address). I'm using Centos 7 so I created the file /etc/sysconfig/network-scripts/ifcfg-eth0:1 with the following:

DEVICE=eth0:1
IPADDR=10.10.10.100
NETMASK=255.255.255.255

For some reason I couldn't say PREFIX=32, not sure why. Anyway, you'll then have a multihomed machine. Leave the other interface alone as well as the gateway.

Next step is to put up a /32 route on your next-hop gateway of that host. I'm using pfSense there so I first had to add a gateway for the machine itself so I could add the static route. So I went to System/Routing/Gateways and added a new one with 10.20.20.100 as the gateway address and picking the correct interface. Then to add the static route I went to System/Routing/Static Routes and added a route for 10.10.10.100 to the gateway 10.20.20.100.

I'm running OSPF and have it configured to distribute kernel routes (the kind pfSense creates here) so now I have a route on the network and the address 10.10.10.100 is reachable from everywhere except the the original 10.10.10.0/24 segment. Why? Because the normal behavior for a host on that segment is to send an ARP (who is 10.10.10.100?) on that local segment and wait for a MAC address to be returned. This is where you need the help provided by proxy arp.

A router using proxy arp on a segment notices when there is a smaller route (a /32 perhaps?) elsewhere on the network. When it sees that route, it pretends to be that host and answers the ARP request with it's own MAC. The sending host then uses that MAC as the destination for the Ethernet frame and sends it. The router receives it, strips off the layer 2 header and uses the routing table to decide where to send it.

So the last step in this process is to add a proxy arp address by going to Firewall/Virtual IPs/Add. Choose proxy arp as the type and select the correct network segment. Select 'single address' and then put the address 10.10.10.100/32 there. Save it and you're all set.

Caveats I've found are mostly related to the firewall functions. pfSense likes to make everything stateful. So make sure your paths are symmetrical or you'll find packets getting dropped even if you have an 'allow all' rule in place.

Today we use systemd to manage our boot process and do nice things like reducing boot time by parallelizing operations and managing service dependencies. One area that still needs some work is /etc/fstab. We're all familiar with making entries there to mount local disks, remote filesystems and perform bind mounts through fusefs. The fstab is still a supported and convenient way to manage mounts but it isn't used directly anymore. Instead, systemd uses a component to parse the file and build standard unit files that can be handled in the normal systemd way. The parsing process is supposed to pay attention to things like file system hierarchies so that it can produce the correct dependencies so that everything gets mounted as intended during the boot process. That is of course if it worked properly. I found out the hard way that it doesn't; at least not in the combination of mounts I had in my /etc/fstab.

/dev/mapper/ingest--digitalarchives--vg-root/ ext4 errors=remount-ro 0 1
/dev/mapper/ingest--digitalarchives--vg-swap_1 none swap sw 0 0

10.20.40.13:/mnt/tank/unix/home/home nfs defaults 0 0
10.20.40.13:/mnt/tank/sftp /sftp nfs defaults 0 0

/sftp/sftp_user/reports /var/www/drupal7/sites/default/files/private/reports none bind 0 0

 

The fstab had the normal filesystem entries to get the machine up and running including the root slice, swap, etc. There are a few NFS mounts in there and a single bind mount. The bind mount was the one causing the issue; local filesystems and NFS mounts seem to come up fine but the bind mount depended on the NFS mount as it was a subdirectory. The bind mount would execute first and then the NFS mount that overlapped it would hang and force systemd to emergency mode.

Looking at the systemd documentation and it's handling of fstab it seemed that the dependency could manually be stated there as an option. Following examples I tried the following without success.

/sftp/sftp_user/reports /var/www/drupal7/sites/default/files/private/reports none bind,x-systemd.requires=/sftp 0 0

In the end I left the NFS mounts in the fstab and created my own unit file in /etc/systemd/system named var-www-drupal7-sites-default-files-private-reports.mount that looks like this:

[Unit]
Description=Mount for reports
After=remote-fs.target

[Mount]
What=/sftp/sftp_hslc/reports
Where=/var/www/drupal7/sites/default/files/private/reports
Type=none
Options=bind

[Install]
WantedBy=multi-user.target

This makes sure all the NFS file mounting is finished before it tries the bind mount. Now things mount in the proper order and the machine actually boots.

I suspect the problem has to do with the fact that I am using a bind mount. Perhaps the bind mounts are done before remote filesystems without regard to options specified in fstab. I could call it a bug since systemd does not interpret fstab and produce the correct intended result.

These commands were executed in a user context, no sudo needed. The routes added to your VPN connection should survive a reboot.

First I list out my network connections to make sure I have the name correct:

$ networksetup -listallnetworkservices

An asterisk (*) denotes that a network service is disabled.
Ethernet
Wi-Fi
home.mydomain.net
*INT
Bluetooth PAN
Thunderbolt Bridge
COGENT-HSLC
GUEST
COGENT-ACCESSPA
Verizon-DSL
DMZ
MGMT
STORAGE
55 with Default
55
52
52 with Default
HSLC

I want to add a route to home.mydomain.net so I do:

$ networksetup -setadditionalroutes home.mydomain.net 192.168.2.0 255.255.255.0 192.168.1.71

I happen to know that the IP address on the other side of the gateway is 192.168.1.71. If you don't know, connect to your vpn and look in the routing table with:

$ netstat -nr -f inet

If you want more than one route you have to combine them all into one command. You can't add them one at a time as the current route will override the previous. So to add two routes I would do:

$ networksetup -setadditionalroutes home.mydomain.net 192.168.2.0 255.255.255.0 192.168.1.71 192.168.3.0 255.255.255.0 192.168.1.71

If your VPN name has a space in it, put quotes around the name like:

$ networksetup -setadditionalroutes "home vpn" 192.168.2.0 255.255.255.0 192.168.1.71

Finally, to delete all the additional routes just leave the route blank.

$ networksetup -setadditionalroutes "home vpn"

To check your work:

$ networksetup -getadditionalroutes "home vpn"

After searching and finding all the little pieces of this process I thought I would document it here for future referral.

The main issue with moving the VM from one hypervisor platform to the other is that the initial RAM disk doesn't have the Xen network and block device drivers. So even if you can successfully convert the disk through export/import or a third party (QEMU perhaps) application, the machine won't come up.

What's needed is to rebuild the RAM disk with the right drivers and then tell grub to use it.

First, let's build the new RAM disk image. Log into the running VM under VMWare and do the following:

# cd /boot

# ls initramfs*

initramfs-0-rescue-898e9089de1c4bcfac2d751bbd9e7f10.img
initramfs-3.10.0-693.21.1.el7.x86_64.img
initramfs-3.10.0-693.21.1.el7.x86_64kdump.img
initramfs-3.10.0-693.el7.x86_64.img
initramfs-3.10.0-693.el7.x86_64kdump.img
initramfs-3.10.0-862.el7.x86_64.img
initramfs-3.10.0-862.el7.x86_64kdump.img

I'm going to use the newest kernel to build my RAM disk. If you have a different kernel (likely), adjust the name as needed.

# mkinitrd --omit-scsi-modules --with=xennet --with=xenblk --preload=xenblk /boot/initramfs-3.10.0-862.el7.x86_64-XEN.img 3.10.0-862.el7.x86_64

ls -ld initramfs*

-rw-------. 1 root root 51307405 Mar 14 2018 initramfs-0-rescue-898e9089de1c4bcfac2d751bbd9e7f10.img
-rw------- 1 root root 21008864 May 10 2018 initramfs-3.10.0-693.21.1.el7.x86_64.img
-rw------- 1 root root 13336203 May 10 2018 initramfs-3.10.0-693.21.1.el7.x86_64kdump.img
-rw-------. 1 root root 20349498 Mar 14 2018 initramfs-3.10.0-693.el7.x86_64.img
-rw-------. 1 root root 18730293 Mar 14 2018 initramfs-3.10.0-693.el7.x86_64kdump.img
-rw------- 1 root root 21049754 May 10 2018 initramfs-3.10.0-862.el7.x86_64.img
-rw------- 1 root root 12976861 Jul 13 07:33 initramfs-3.10.0-862.el7.x86_64kdump.img
-rw------- 1 root root 21075430 Dec 14 10:29 initramfs-3.10.0-862.el7.x86_64-XEN.img

Okay, you can see the new image and it's about the same size as the other one too.

Next, we need to copy the text from a boot entry that grub uses. Find it in /boot/grub2/grub.cfg. Don't edit this file, just find the section that looks like this:

menuentry 'CentOS Linux (3.10.0-862.el7.x86_64) 7 (Core)' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-693.el7.x86_64-advanced-0a5149be-1fe5-40bf-8597-c0c588af88c8' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_msdos
insmod xfs
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1' efa02a93-827e-4734-8a44-7eb1ef93119a
else
search --no-floppy --fs-uuid --set=root efa02a93-827e-4734-8a44-7eb1ef93119a
fi
linux16 /vmlinuz-3.10.0-862.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8
initrd16 /initramfs-3.10.0-862.el7.x86_64.img
}

Take that text and add it to the end of the file in /etc/grub.d/40_custom. You then want to change two things. First change the name of the entry so you 'll be able to see it when you boot. Then change the name of the RAM disk image being used. My finished file looks like this:

#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
menuentry 'CentOS Linux (3.10.0-862.el7.x86_64) 7 (Core) with xen' --class centos --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-693.el7.x86_64-advanced-0a5149be-1fe5-40bf-8597-c0c588af88c8' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_msdos
insmod xfs
set root='hd0,msdos1'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 --hint='hd0,msdos1' efa02a93-827e-4734-8a44-7eb
1ef93119a
else
search --no-floppy --fs-uuid --set=root efa02a93-827e-4734-8a44-7eb1ef93119a
fi
linux16 /vmlinuz-3.10.0-862.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet LANG=en_US.UTF-8
initrd16 /initramfs-3.10.0-862.el7.x86_64-XEN.img
}

Lastly, we need to rebuild the grub menu.

# grub2-mkconfig --output=/boot/grub2/grub.cfg

That should do it. Now you can export your machine and when it boots under Xenserver make sure you choose the right boot option when it starts up. You can make that the default option in grub if you want to.

I did not have any issue using the RAM disk containing the xen drivers under VMWare either so I'll bet you could just copy the file over the other RAM disk and it would boot under either OS. May try that just to make sure it works okay.

Easiest way to get your network up and running correctly and quickly:

1. Make sure you configure your network with static IP address (if needed) when installing. Enter the IP address, network, gateway, etc.

2. Let VMWare assign the MAC. Much easier than trying to manage it through udev or the config files.

Your network configuration file is in /etc/sysconfig/network-scripts/ directory. You only need to touch the file for the specific adapter that is created (e.g. ifcfg-ens192).

If you add another adapter to the machine in VMWare use the following steps:

1. I usually reboot to make sure it is properly detected by the OS.

2. Do an ifconfig -a to see the new adapter name.

3. Copy the ifcfg-ensnnn file to the new name, then edit the contents. Go over each item carefully. Make sure you change the DEVICE line to the new name. Change the UUID line. To get the UUID you can:

nmcli -t --fields uuid,device con show

4. Reboot to make sure it all works, your adapter should be up and have the right IP address, etc. Use ifconfig -a to check it.

To add a static route:

Create a file in /etc/sysconfig/network-scripts called route-ens192 (or whatever the adapter is).

Put the route parameters in there like this:

10.0.0.0/8 via 10.182.3.1 dev ens192

Reboot, then check netstat -nr to make sure you see the proper kernel route.