LACP stands for Link Aggregation Control Protocol. It’s a network protocol used to combine multiple physical network links into a single logical link to increase bandwidth and provide redundancy. It’s part of the IEEE 802.3ad standard (now 802.1AX).
Here’s a breakdown of what it does and why it’s useful:
- Increases Bandwidth
- By bundling multiple links (like two or more Ethernet cables) between switches or between a switch and a server, the total throughput can be higher than a single link.
- Provides Redundancy
- If one physical link fails, traffic is automatically rerouted over the remaining links, so the connection stays up.
- Dynamic Configuration
- LACP allows devices to automatically detect and configure link aggregation groups, making it easier to manage than static link aggregation.
- Load Balancing
- Traffic can be distributed across the aggregated links based on rules like source/destination IP, MAC addresses, or TCP/UDP ports.
Let's configure and test it in my homelab.
Homelab
In my homelab, I have Force10 S60 L3 switch witch supports LACP. I will aggregate port GigabitEthernet 0/28 and GigabitEthernet 0/38 into LAG (Link Aggregation Group) Port-channel 1. I use VLAN 4 for datacenter management with IP subnet 192.168.4.0/24 and Force10 switch act as a default gateway with IP address 192.168.4.254. For FreeBSD 14.3 server, I will use IP address 192.168.4.124.
Below is connectivity schema between FreeBSD Server and Force10 Switch in my homelab.
 |
| FreeBSD Server and Force10 Switch Connectivity Schema |
Switch configuration
I will demonstrate configuration only on GigabitEthernet 0/38, but it must be done on all physical interfaces participating in port-channel (GigabitEthernet 0/28 in my case). First of all, all physical interfaces participating in port-channel must have clear (empty) configuration as documented below ...
f10-s60(conf-if-gi-0/38)#show conf! interface GigabitEthernet 0/38 description BHV01-nic1 no ip address no shutdown f10-s60(conf-if-gi-0/38)#
When physical interface has empty configuration, we can assign that physical interface (gi 0/38) into port-channel 1. The desired configuration is documented below ...
f10-s60(conf-if-gi-0/38)#show conf! interface GigabitEthernet 0/38 description BHV01-nic1 no ip address ! port-channel-protocol LACP port-channel 1 mode active no shutdown f10-s60(conf-if-gi-0/38)#
Now we should see port-channel 1.
f10-s60#sh interfaces port-channel briefCodes: L - LACP Port-channel LAG Mode Status Uptime Ports L 1 L3 down 00:00:00 f10-s60#
f10-s60#conf f10-s60(conf)#int port-channel 1 f10-s60(conf-if-po-1)#show conf! interface Port-channel 1 no ip address no shutdown f10-s60(conf-if-po-1)#
We have port-channel 1, and it is in down state and without detailed configuration. Let's set MTU to 9252, enable switchport mode, configure spanning-tree and enable the virtual port-channel interface. We should have configuration documented below ...
f10-s60(conf-if-po-1)#show conf! interface Port-channel 1 no ip address mtu 9216 switchport spanning-tree rstp edge-port no shutdown f10-s60(conf-if-po-1)#
Do not forget, that VLAN(s) must be allowed for particular port-channel. Let's configure VLAN 4 as a native (untagged) VLAN as it is used for datacenter management ...
f10-s60(conf-if-vl-4)#show conf! interface Vlan 4 description DC-MGMT ip address 192.168.4.254/24 untagged GigabitEthernet 0/2-4,6,10,16-19,23,25-27,35-37,43 untagged Port-channel 1 no shutdown f10-s60(conf-if-vl-4)#
If you need other VLANs (iSCSI, NFS, various VM traffics, etc.) you can add them later as tagged VLANs levering 802.1Q tagging.
FreeBSD LACP Configuration
This is the LACP configuration in /etc/rc.conf
ifconfig_bge3="mtu 9000 up"
cloned_interfaces="lagg0"
Physical interfaces (bge2,bge3) must be configured for Jumbo Frames (MTU 9000) and on the other side (Force10 Switch) must be Jumbo Frames enabled as well (MTU 9216).
lagg0 is cloned virtual interface aggregating bge2 and bge3.
Check LACP status
LACP has few properties which should be validated, understand and properly configured.
- LACP timers (fast, slow)
- Load balancing hash algorithms
Let's check LACP Port-channel status
f10-s60#sh interfaces po1 briefCodes: L - LACP Port-channel LAG Mode Status Uptime Ports L 1 L2L3 up 00:18:41 Gi 0/28 (Up) Gi 0/38 (Up) f10-s60#sh interfaces po1 descrInterface OK Status Protocol Description Port-channel 1 YES up up BHYVE01 f10-s60#
We can see that both ports (Gi 0/28, Gi 0/38) participating in port-channel 1 and are up and running.
A port-channel with the L2L3 mode is a versatile interface that can perform two distinct networking functions simultaneously.
- Layer 2 (L2) VLAN Trunking: The port-channel can be configured to carry traffic for one or more VLANs. You would typically use the switchport mode trunk and switchport trunk allowed vlan commands to specify which VLANs are permitted to traverse the aggregated link. This is the L2 part of the configuration.
- Layer 3 (L3) IP Routing: The port-channel can also have an IP address directly assigned to it. This allows the switch to act as a router for that link, enabling the port-channel to be used as a routed uplink or a termination point for a specific subnet. This is the L3 part.
LAG 1 (Port-channel 1) is up with both physical ports (Gi 0/28, Gi 0/38) up and running. Port-channel 1 state is OK. That's perfect, so let's focus on LACP timers, LACP hash algorithms, and Jumbo Frames (MTU 9000/9216).
LACP with "fast" or "slow" timers?
You can configure LACP with "fast" or "slow" timers, which dictates how frequently LACPDU (LACP Data Unit) packets are sent between the LACP-enabled devices. This is not a direct setting on the FreeBSD lagg interface itself, but rather a property of the LACP protocol that is typically configured on the switch.
Force10 Switch
It seems that it is not configurable on my Force10 S60 or I did not find the way how to configure it. But I can check LACP timers there by following command ...
f10-s60#sh lacp 1Port-channel 1 admin up, oper up, mode lacp Actor System ID: Priority 32768, Address 0001.e896.0203 Partner System ID: Priority 32768, Address 90b1.1c12.e751 Actor Admin Key 1, Oper Key 1, Partner Oper Key 203 LACP LAG 1 is an aggregatable link A - Active LACP, B - Passive LACP, C - Short Timeout, D - Long Timeout E - Aggregatable Link, F - Individual Link, G - IN_SYNC, H - OUT_OF_SYNC I - Collection enabled, J - Collection disabled, K - Distribution enabled L - Distribution disabled, M - Partner Defaulted, N - Partner Non-defaulted, O - Receiver is in expired state, P - Receiver is not in expired state Port Gi 0/38 is enabled, LACP is enabled and mode is lacp Actor Admin: State ACEHJLMP Key 1 Priority 32768 Oper: State ACEGIKNP Key 1 Priority 32768 Partner Admin: State BDFHJLMP Key 0 Priority 0 Oper: State ADEGIKNP Key 203 Priority 32768 f10-s60#
Actor (Force10 S60) is configured and operates with Short Timeout (aka Fast LACP timers).
Partner (FreeBSD Server) is configured and operates with Long Timeout (aka Slow LACP timers).
Both sides should use the same timers, and Fast LACPDU (Short Timeout) is recommended.
FreeBSD Server
FreeBSD could be configured to support Fast LACPDU (Short Timeout) by following ifconfig options
- lacp_fast_timeout
- Enable lacp fast-timeout on the interface.
- lacp_fast_timeout
- Disable lacp fast-timeout on the interface.
That's why we have this in /etc/rc.conf
ifconfig_lagg0="laggproto lacp laggport bge2 laggport bge3 lagghash l2,l3,l4 lacp_fast_timeout"
When lacp_fast_timeout is set, Force10 Switch reports partner (FreeBSD Server) as C (Short Timeout).
f10-s60#sh lacp 1Port-channel 1 admin up, oper up, mode lacp Actor System ID: Priority 32768, Address 0001.e896.0203 Partner System ID: Priority 32768, Address 90b1.1c12.e751 Actor Admin Key 1, Oper Key 1, Partner Oper Key 203 LACP LAG 1 is an aggregatable link A - Active LACP, B - Passive LACP, C - Short Timeout, D - Long Timeout E - Aggregatable Link, F - Individual Link, G - IN_SYNC, H - OUT_OF_SYNC I - Collection enabled, J - Collection disabled, K - Distribution enabled L - Distribution disabled, M - Partner Defaulted, N - Partner Non-defaulted, O - Receiver is in expired state, P - Receiver is not in expired state Port Gi 0/28 is enabled, LACP is enabled and mode is lacp Actor Admin: State ACEHJLMP Key 1 Priority 32768 Oper: State ACEGIKNP Key 1 Priority 32768 Partner Admin: State BDFHJLMP Key 0 Priority 0 Oper: State ACEGIKNP Key 203 Priority 32768 Port Gi 0/38 is enabled, LACP is enabled and mode is lacp Actor Admin: State ACEHJLMP Key 1 Priority 32768 Oper: State ACEGIKNP Key 1 Priority 32768 Partner Admin: State BDFHJLMP Key 0 Priority 0 Oper: State ACEGIKNP Key 203 Priority 32768 f10-s60#
I do not know
- why Admin State is D (Long Timeout)
- how to show LACP timeout on FreeBSD server side
but based on Force10 Switch status, it seems that both sides use Fast LACPDU (Short Timeout).
LACP Hash Algorithms
FreeBSD Server
On FreeBSD Server, the hash algorithm is configured on the lagg0 interface, not the physical member ports. Particular hash algorithm can be configured by adding the lagghash option to ifconfig_lagg0 line in /etc/rc.conf.
ifconfig_lagg0="laggproto lacp laggport bge2 laggport bge3 lagghash l2,l3,l4"
We have following options
- l2: Layer 2 hashing (based on MAC addresses)
- l3: Layer 3 hashing (based on IP addresses)
- l4: Layer 4 hashing (based on TCP/UDP ports)
- l2,l3: Combines Layer 2 and Layer 3 hashing
- l2,l3,l4: Combines Layer 2, Layer 3, and Layer 4 hashing.
- This is the most common and effective algorithm as it provides the most granular distribution
- l2,l3,l4 is the default option when lagghash option is omitted
Below is the output from FreeBSD Server in my lab ...
root@bhyve01:~ #ifconfig lagg0lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 9000 options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE> ether 90:b1:1c:12:e7:51 hwaddr 00:00:00:00:00:00 inet 192.168.4.124 netmask 0xffffff00 broadcast 192.168.4.255 laggproto lacp lagghash l2,l3,l4 laggport: bge2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> laggport: bge3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> groups: lagg media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> root@bhyve01:~ #
Force10 Switch
I did not find the way, how to configure or view LAG hash algorithm on Force10 Switch.
We can clear counter statistics and observe Input/Output statistics on physical interfaces.
f10-s60#clear countersClear counters on all interfaces [confirm] f10-s60#show interfaces gigabitethernet 0/38GigabitEthernet 0/38 is up, line protocol is up Port is part of Port-channel 1 Description: BHV01-nic1 Hardware is DellForce10Eth, address is 00:01:e8:96:02:05 Current address is 00:01:e8:96:02:05 Pluggable media not present Interface index is 43828226 Internet address is not set MTU 9216 bytes, IP MTU 9198 bytes LineSpeed 1000 Mbit, Mode full duplex Auto-mdix enabled, Flowcontrol rx off tx off ARP type: ARPA, ARP Timeout 04:00:00 Last clearing of "show interface" counters 00:00:03 Queueing strategy: fifo Input Statistics: 3 packets, 384 bytes 0 64-byte pkts, 0 over 64-byte pkts, 3 over 127-byte pkts 0 over 255-byte pkts, 0 over 511-byte pkts, 0 over 1023-byte pkts 3 Multicasts, 0 Broadcasts 0 runts, 0 giants, 0 throttles 0 CRC, 0 overrun, 0 discarded Output Statistics: 1 packets, 64 bytes, 0 underruns 1 64-byte pkts, 0 over 64-byte pkts, 0 over 127-byte pkts 0 over 255-byte pkts, 0 over 511-byte pkts, 0 over 1023-byte pkts 0 Multicasts, 1 Broadcasts, 0 Unicasts 0 throttles, 0 discarded, 0 collisions Rate info (interval 299 seconds): Input 00.00 Mbits/sec, 0 packets/sec, 0.00% of line-rate Output 00.00 Mbits/sec, 0 packets/sec, 0.00% of line-rate Time since last interface status change: 00:48:38 f10-s60#show interfaces gigabitethernet 0/28GigabitEthernet 0/28 is up, line protocol is up Port is part of Port-channel 1 Description: BHV01-nic2 Hardware is DellForce10Eth, address is 00:01:e8:96:02:05 Current address is 00:01:e8:96:02:05 Pluggable media not present Interface index is 41206786 Internet address is not set MTU 9216 bytes, IP MTU 9198 bytes LineSpeed 1000 Mbit, Mode full duplex Auto-mdix enabled, Flowcontrol rx off tx off ARP type: ARPA, ARP Timeout 04:00:00 Last clearing of "show interface" counters 00:02:35 Queueing strategy: fifo Input Statistics: 155 packets, 20287 bytes 0 64-byte pkts, 4 over 64-byte pkts, 151 over 127-byte pkts 0 over 255-byte pkts, 0 over 511-byte pkts, 0 over 1023-byte pkts 151 Multicasts, 0 Broadcasts 0 runts, 0 giants, 0 throttles 0 CRC, 0 overrun, 0 discarded Output Statistics: 200 packets, 15283 bytes, 0 underruns 185 64-byte pkts, 8 over 64-byte pkts, 7 over 127-byte pkts 5 over 255-byte pkts, 0 over 511-byte pkts, 0 over 1023-byte pkts 81 Multicasts, 114 Broadcasts, 5 Unicasts 0 throttles, 0 discarded, 0 collisions Rate info (interval 299 seconds): Input 00.00 Mbits/sec, 0 packets/sec, 0.00% of line-rate Output 00.00 Mbits/sec, 0 packets/sec, 0.00% of line-rate Time since last interface status change: 00:51:07 f10-s60#
JumboFrames (MTU 9000/9216)
I would like to use FreeBSD server as Virtualization and Storage Server. Jumbo Frames (MTU 9216 on switch, MTU 9000 on server) has performance benefit because less frame segmentation means less CPU cycles and higher network throughput.
In Force10 switch (Dell Networking), we set the MTU 9216 on the port-channel interface, and also on individual physical interfaces.
f10-s60#show running-config interface port-channel 1! interface Port-channel 1 description BHYVE01 no ip address mtu 9216 switchport spanning-tree rstp edge-port no shutdown f10-s60#show running-config interface gigabitethernet 0/28! interface GigabitEthernet 0/28 description BHV01-nic2 no ip address mtu 9216 ! port-channel-protocol LACP port-channel 1 mode active no shutdown f10-s60#show running-config interface gigabitethernet 0/38! interface GigabitEthernet 0/38 description BHV01-nic1 no ip address mtu 9216 ! port-channel-protocol LACP port-channel 1 mode active no shutdown f10-s60#
In FreeBSD server, we set the MTU on the physical interfaces (bge2, bge3) that are members of the LACP bond. The lagg0 virtual interface will inherit the MTU from its member ports. It is stored in /etc/rc.conf
ifconfig_bge2="mtu 9000 up"
ifconfig_bge3="mtu 9000 up"
Correct configuration of Jumbo Frames (MTU 9000) can be tested by following command ...
root@bhyve01:~ #ping -s 8972 -D 192.168.4.254PING 192.168.4.254 (192.168.4.254): 8972 data bytes 8980 bytes from 192.168.4.254: icmp_seq=0 ttl=255 time=1.605 ms 8980 bytes from 192.168.4.254: icmp_seq=1 ttl=255 time=1.385 ms 8980 bytes from 192.168.4.254: icmp_seq=2 ttl=255 time=1.329 ms ^C --- 192.168.4.254 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1.329/1.440/1.605/0.119 ms root@bhyve01:~ #
Bigger frame should not be transferred ...
root@bhyve01:~ #ping -s 8973 -D 192.168.4.254PING 192.168.4.254 (192.168.4.254): 8973 data bytes ping: sendto: Message too long ping: sendto: Message too long ping: sendto: Message too long ^C --- 192.168.4.254 ping statistics --- 3 packets transmitted, 0 packets received, 100.0% packet loss root@bhyve01:~ #
Conclusion
LACP has positive impact on availability (two links in port-channel), performance (aggregated bandwidth with load-balancing based on defined hash), and manageability (configuration can be done on single virtual interface Po1 instead of multiple physical interfaces).
To be honest, in my homelab the availability is limited, because both links (bge2,bge3) are connected to single Force10 Switch because I do not have two switches in my homelab, and in case of two switches, these switches must support some kind of MLAG (Multi-Chassis Link Aggregation) like Cisco Virtual Port-Channel (vPC), Force10 Virtual Link Trunking (VLT), etc.
However, it is crucial to have both ends (Server <-> Switch) under control and fully understand and test how LACP works.
No comments:
Post a Comment