Thursday, January 5, 2012

WCCPv2 and Websense Content Gateway–Configuration

I have completed a configuration write-up for WCCPv2 and the Websense Content Gateway. I go over basic configuration, IP Spoofing, and lightly touch on the differences involving using IOS Routers, Switches, and the ASA.  I also list some best-practice setups and gotchas.  Below is a very basic copy and paste of the document; it is missing pictures and formatting. You can download the full document in PDF format HERE. Enjoy!

Preface

This is an unofficial document. This document is presented as-is with no warranty involved in its accuracy of information and content. While I have been able to test and verify some of the content, it is mostly based on what “should” happen in theory and my own research. Unfortunately I do not have the environment available to test every situation.

Lastly, it is assumed you have read and are familiar with the terms and methods described in Module 1: Overview, so that information will not be re-hashed.

Planning

WCCP can be very simple and very complex at the same time. It is very important that you treat it like any other network implementation and take the time to analyze your network environment and its capabilities and limitations. Supported features can differ greatly depending on hardware/software platform, so any specific questions regarding the capabilities of a customer’s equipment should be identified and directed to Cisco during the planning phase instead of being forced to do so during troubleshooting, as this can cause delays in implementation, unnecessary outages, and even unplanned financial ramifications in the case where a software upgrade is needed.

Configure WCCP Server – IOS Router/Switch

For this example, we will configure WCCP for the given network topology below:

Create standard ACL to be used for the Group List (Optional)

In this step, we will configure an ACL to be utilized as a “group list”. This is optional, but a best practice security measure. This specifies what is allowed to participate in a given service group. Without this, any device could leverage WCCP and send Here I Am (HIA) messages and join the service group as long as it meets all requirements and includes the password if configured. In a worst case scenario, an attacker could leverage this to become the designated web cache, and then send a redirect assignment to effectively force the WCCP server to send all traffic to his machine as a man-in-the-middle attack.

In the output provided below, an ACL is configured to only permit 10.212.8.11:

! First, we configure

1711-2Router#conf t

Enter configuration commands, one per line. End with CNTL/Z.

1711-2Router(config)#ip access-list standard wccp_group_list

11711-2Router(config-std-nacl)#permit host 10.212.8.11

711-2Router(config-std-nacl)#deny any

1711-2Router(config-std-nacl)#exit

! Then, we verify

1711-2Router(config)#do sh ip access-list wccp_group_list

Standard IP access list wccp_group_list

10 permit 10.212.8.11

20 deny any[1]

Create extended ACL to be used for the Redirect List (Optional)

In the next step, we will configure an extended ACL to use for the “redirect list”. This identifies the interesting traffic that we want to redirect. In this case, we want to redirect only a part of the 10.212.0.0/16 network; specifically, IP traffic sourced from the 10.212.8.8/29 subnet, and nothing else.

! First, we configure

1711-2Router(config)#ip access-list extended wccp_redirect_list

1711-2Router(config-ext-nacl)#permit ip 10.212.8.8 0.0.0.7 any

1711-2Router(config-ext-nacl)#deny ip any any

1711-2Router(config-ext-nacl)#exit

! Then, verify

1711-2Router(config)#do sh ip access-list wccp_redirect_list

Extended IP access list wccp_redirect_list

10 permit ip 10.212.8.8 0.0.0.7 any

20 deny ip any any

Enable WCCP

Next, we enable WCCP on the router. We’re going to utilize service ID 0 for the service group, and a password for added security. Until we configure and enable WCCP on the Websense Content Gateway (WCG), the service group will not establish.

! Enable WCCP for a given service ID # and assign the associated redirect and group lists. Optionally, you can also specify a password.

1711-2Router(config)#ip wccp 0 redirect-list wccp_redirect_list group-list wccp_group_list password websense

! Verify

1711-2Router(config)#do sh ip wccp

Global WCCP information:

Router information:

Router Identifier: -not yet determined-

Protocol Version: 2.0

Service Identifier: 0

Number of Cache Engines: 0

Number of routers: 0

Total Packets Redirected: 0

Redirect access-list: wccp_redirect_list

Total Packets Denied Redirect: 0

Total Packets Unassigned: 0

Group access-list: wccp_group_list

Total Messages Denied to Group: 9

Total Authentication failures: 1

Total Bypassed Packets Received: 0

Enable WCCP on the Interface

The last part of the configuration on the router will be to enable redirection on the desired interface. For this example, all Internet traffic is coming in on the SVI for VLAN 10 (10.212.0.4). This is where we will configure WCCP redirection.

! Configure WCCP redirection on the desired interface

1711-2Router(config)#int vl 10

1711-2Router(config-if)#ip wccp 0 redirect in[2]

1711-2Router(config-if)#exit

! Verify configuration

1711-2Router(config)#do sh run int vl 10

Building configuration...

Current configuration : 151 bytes

!

interface Vlan10

ip address 10.212.0.4 255.255.0.0

ip access-group permit_all in

ip wccp 0 redirect in

ip nat inside

ip virtual-reassembly

end

Configure the WCG[3]

Moving on to the WCG, after WCCP is enabled, go to Configure > Networking > WCCP in the WCG Manager and edit the file to set up the service group as shown below. In this example, we are using dynamic service ID 0 for both HTTP and HTTPS[4].

Once this is completed and the WCG is restarted, it will start sending HIAs to the WCCP Server and the service group will establish.

Verify establishment of Service Group

It is important to verify the establishment of the service group before expecting traffic to be redirected. We can also take a look at the assignment method information as well as the router’s cluster view.

1711-2Router#sh ip wccp 0

Global WCCP information:

Router information:

Router Identifier: 10.212.0.4

Protocol Version: 2.0

Service Identifier: 0

Number of Cache Engines: 1

Number of routers: 1

Total Packets Redirected: 0

Redirect access-list: wccp_redirect_list

Total Packets Denied Redirect: 0

Total Packets Unassigned: 0

Group access-list: wccp_group_list

Total Messages Denied to Group: 4543

Total Authentication failures: 739

Total Bypassed Packets Received: 0

1711-2Router#sh ip wccp 0 det[5]

WCCP Cache-Engine information:

Web Cache ID: 10.212.8.11

Protocol Version: 2.0

State: Usable

Initial Hash Info: 00000000000000000000000000000000

00000000000000000000000000000000

Assigned Hash Info: FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

Hash Allotment: 256 (100.00%)

Packets Redirected: 0

Connect Time: 00:00:53

Bypassed Packets

Process: 0

Fast: 0

CEF: 0

1711-2Router#sh ip wccp 0 view

WCCP Routers Informed of:

10.212.0.4

WCCP Cache Engines Visible:

10.212.8.11

WCCP Cache Engines NOT Visible:

-none-

For additional verification, you can also verify establishment of the service group within the WCG Manager. Notice that it specifies a configured mode, and a negotiated mode.

Configure and Verify Filtering

Once the service group has been established, we can reasonably expect traffic to be redirected to the WCG. The easiest way to verify this is to simply setup a policy to block traffic, and confirm you can pull a block page.

The Router ID: More than Just a Number

In the past, I have thought of the router ID as mostly just cosmetic. Over time however, I’ve found this to be very important to be aware of, for at least two major reasons.

Instability within the service group can occur if you have two routers in the same service group that happen to have the same Router ID. It’s difficult to illustrate this, but as described in the previous Overview Module, the WCCP Server sends I See You (ISU) messages to its WCG(s) with a sequential Receive ID, and it expects that same Receive ID to be communicated back to it. If the WCG receives two different Receive IDs, it will attempt to communicate the last receive ID it received back to the routers in its next HIA message. This can cause those HIAs to be dropped by one of the routers due to the Receive ID being invalid.

This condition can be identified in debug output on the router, but that will be saved for the upcoming troubleshooting module. This is easily fixed by adding a loopback interface on one of the routers with a higher IP address so that a different Router ID is acquired.

Another major concept to be aware of is that when utilizing GRE, the Router ID is used as the source IP when the initial redirection is performed, so the packet on a GRE forward looks something like this:

What this means is that you can effectively prevent this forward from occurring by having an ACL on an interface along the path this traffic takes that doesn’t permit the IP address of the Router ID, even though it explicitly allows the IP of the interface where WCCP redirection is enabled.

Making Changes to Established Service Groups

If a change is needed in the redirect list or in other areas of an active service group, it is considered a best practice to make those changes in the following order:

1. Disable WCCP on all affected WCGs

2. Remove interface configuration

3. Remove or change the global configuration

a. Redirect/Group Lists on WCCP Server

b. Forward/Return/Assignment Method on WCG

4. Apply new global configuration

5. Re-register WCGs

However, saying this, I have seen changes made to the redirect list ACL on-the-fly with no ill results many times.

IP Spoofing[6]

Enabling IP Spoofing on the WCG presents a unique challenge. In order for this to work correctly, you must control both sides of the Internet traffic; return traffic from the origin server must be redirected back to the WCG to serve to the client.

This is accomplished with a standard service group that keys off of destination port to handle the client Internet traffic, and a “reverse” service group that keys off source port to handle the return traffic from the origin server. Because IP Spoofing causes the WCG to send its traffic with the source IP of the client, this also requires that the WCG be on a different subnet/VLAN than the client traffic in order to prevent the WCCP Server from redirecting traffic sourced from the WCG back to itself.

As shown below, the redirect lists used for these service groups are mirrored. On the standard service group we are redirecting IP traffic sourced from the 10.212.8.8/29 subnet destined anywhere. On the reverse service group, we are redirecting IP traffic destined to the 10.212.8.8/29 subnet sourced from anywhere.

On the WCG, a reverse service group is specified under Advanced Settings. Also, change the assignment method to distribute traffic based off of source IP instead of destination IP.

Case Study – IP Spoofing Gone Wrong

A customer desired to utilize IP Spoofing in his environment. The general traffic flow from the internal LAN to the Internet was through the core switch fabric, up to the router, and then back through the core out to an edge firewall before finally getting to the Internet.

The problem was that the WCG resided in the same subnet/VLAN as his client traffic, resulting in the WCG’s traffic taking the same path to get out to the Internet. There wasn’t an issue with the initial redirection (blue path numbered 1), but when the WCG proxies the request on behalf of the client (red path numbered 2), this IP traffic hit the interface that WCCP redirection was configured on, and due to IP Spoofing, the source IP was that of the client and not the WCG.

Normally a WCCP Server knows not to send traffic from one of its WCCP Clients back to itself due to it keeping a record of its client’s IP addresses in the cluster view, but since the source IP of the client was maintained instead, it matched what was specified in the configured redirect list of the router, and the router sent the traffic back to the WCG (red path numbered 3).

To address this, a separate logical sub-interface was configured (fa 0/0.2) into a new VLAN 20, and the WCG was assigned a new IP address within this new VLAN. This enabled the WCG to take a separate path out to the Internet that was not through the interface where WCCP redirection was configured, as shown below.

Differences with the Cisco Switch

Thankfully there is not much difference in syntax between the IOS router and switch[7] when it comes to configuration. On a switch, you want to leverage the hardware as much as possible. This means:

· Inbound Redirection

· Utilize Mask assignment

· Use L2 Forward/Return

o Note: This requires L2 adjacency between the proxy and the WCCP Server.

Differences with the Cisco ASA

The ASA is a beast all of its own. This is probably where I see the most of the TS cases if I had to guess. Personally, I’d rather leverage WCCP elsewhere if I could, but I have seen it deployed successfully in many environments. Some of the limitations include:

· Cannot use IP Spoofing

· Cannot redirect traffic from one security zone onto another (DMZ to Inside, etc.)

· Cannot utilize bypasses in the WCG, as this will cause a WCCP redirect loop

For configuration:

· Utilize GRE Forward/Return

· Utilize Hash Assignment

· Use specific Layer 4 statements in the redirect list ACL

o Examples:

§ Good: permit tcp 10.212.8.8 255.255.255.248 any eq www

§ Bad: permit ip 10.212.8.8 255.255.255.248 any


[1] There is an implied “deny any” at the end of any ACL. Configuring this deny statement is to help with visualizing it.

[2] Inbound redirection is recommended wherever possible to reduce processor overhead.

[3] It should be noted that the version of the WCG used for this document is v7.6.2. Our implementation of WCCP was heavily improved upon in version 7.6.x in comparison to v7.5. This allows us to specify multiple ports for a single service ID.

[4] Customers often read Cisco documentation and attempt to use the well-known service ID of “web-cache”. The WCCP Server will accept this, however the WCG utilizes dynamic service IDs. This will cause the HIA messages sent by the WCG to be dropped and the service group will not establish.

[5] Depending on your hardware/software platform, this output can also show what methods were ultimately negotiated. Saying that, it is important to remember that what you specify in the WCG doesn’t guarantee that method is supported by the WCCP Server and/or will be negotiated.

[6] IP Spoofing with HTTPS is only supported in v7.6 but has known performance issues. This has been reported to be rectified in v7.6.2

[7] Something to note is the Nexus series of switches. At the time of this writing, I have no experience with this and the NX-OS, so I have purposely left this out. As always, refer to your Cisco documentation and representative for specific information regarding this platform.