Friday, January 4, 2013

Cisco Multicast Expansion Table (MET) is Full

Today, some users aren't able to join multicast groups to watch their favorite video. After pinpointing to the upstream Cisco 7600 router, I noticed the following error message in the log:

 %MCAST-SP-4-MET_THRESHOLD_EXCEEDED: Multicast Expansion table has exceeded 98% of its capacity and is reaching its maximum
%MMLS-SP-6-MET_LIMIT_EXCEEDED: Failed to allocate MET entry, exceeded system limit of (32744) entries. Number of times MET limit is exceeded in the last 1 min : 1
%CONST_MFIB_LC-SP-6-MET_MCAST_ALLOC_FAILURE: Failed to allocate MET entries for IP multicast entry (S:*.*.*.*, G:*.*.*.*)

The router is configured PIM in sparse mode and an RP manually set.

(hostname)#show platform hardware capacity multicast
L3 Multicast Resources
  Replication mode: egress
  Replication capability: Module                              Mode
                          2                                                 Egress    
                          3                                                 Egress    
                          4                                                 Egress    
                          5                                                 Egress    
  MET table Entries: Module                             Total    Used    %Used
                     3                                                65526       28        1%
                     4                                                65526       38        1%
                     5                                                32744   32744     100%

Look like the Multicast Expansion Table (MET) is fully utilized. In Cisco multicast architecture, MET is populated with Output Interface Lists (OILs e.g. VLAN, routed interfaces) that requires packets to be replicated over. For example: (S,G) with three OIFs, multicast replication creates three copies of every packet received from source (S) destined to group (G). When MET is full, new users won't be able to join the igmp group, as their (S,G) and connected interface won't be added in the table.

There doesn't seem to be a way to clear MET entries except to reload the router. Cisco has a pdf guide that is useful for better understanding and troubleshooting of multicast in Catalyst 6500. The guide did mention about monitoring of MET utilization but short on the resolution.

After-note: Packet replication can be performed either egress (default) or ingress. Strangely, after changing the direction (not rebooting), the clients are able to join the IGMP group. The direction can be changed using:
(config)# mls ip multicast replication-mode ingress or the newer command (IOS 15.x)
(config)# ip multicast hardware-switching replication-mode ingress

Update:
According to Cisco TAC, there was a known bug (CSCtg53299) whereby MET entries are not deleted properly, which in turn, led to the filling up of MET table. A solution is to upgrade the firmware to version 15.1(01)S1 and above.

4 comments:

  1. Just curious as I've never ran into this issue- How many users do you think you had running multicast at that time?

    ReplyDelete
    Replies
    1. Multicast PIM is designed to replicate the same packet stream on interfaces subscribing to the same group. You may have 1000 users on one VLAN or just 10 users on 10 VLAN interfaces. Hence, the number of users is really immaterial.

      Delete
  2. What was done in troubleshooting to show which are the users that aren't able to join multicast groups?

    ReplyDelete
  3. Bardzo fajnie napisane. Jestem pod wrażeniem i pozdrawiam.

    ReplyDelete