Implementing IPSec

Now that you’re familiar with some of the property settings, lets configure a new rule for accessing the HR server, as the name I gave the policy suggests. This particular rule will require transport mode, since I don’t want to define any type of tunnel. I also should have some concept of which port or ports I need to secure for a particular application if applicable. For my example, I am going to assume that we want all IP traffic between this client and my HR server encrypted. You should experiment with more granular policies on your own.

Clicking the Add button on the rules tab opens the Create IP Security Rule Wizard.

The first step is deciding whether or not the rule specifies a tunnel. If not, the system will be configured to use transport mode.

The Network type screen comes next. From here we can decide whether you want the rule to be used for all network connections, only LAN, or only Remote access. I’ve chosen that it should apply to all.

It may seem strange that you are again presented with the authentication mechanism screen again after this, but remember that each rule can have its own. The one that we specified earlier was the authentication mechanism for the default response rule. I’ll skip the screen shot since it’s exactly the same as the one presented previously.

The next screen is where the going gets tough, since you need to specify the exact types of traffic that you wish to secure. You might be able to escape easily by choosing to secure all IP and ICMP traffic, but I suggest taking a look at adding a rule that meets your needs a little better.

Click the Add button to add a new filter list.

Things get even more complex since you can add multiple filters using this tool. If you click the Add button, you are moved into yet another wizard that will walk you through the process of creating a filter. I am going to create only one (note the descriptive name), that specifies that all of the computers on a given subnet must use IPSec to communicate with the HR Server.

After choosing the traffic source (my subnet), I must next choose the traffic destination.

I have chose the IP address of my HR server, Next step is choosing the IP protocol type to be included in my filter. I am going to choose ANY, although I could be more specific and choose a given TCP port or protocol ID if required.

The next screen allows you to edit your filter without the wizard. Remember that the policy is not yet complete; we must choose to enable our new rule in the policy by selecting it.

The next step is to actually specify the filter actions. What will happen when a user does something that meets rules of the filter? Remember that we haven’t yet said whether AH or ESP should be used, whether these packets should be allowed or denied, and so forth. I know this is a long process, but we’re almost there.

The Filter Action screen allows us to control how this will work. I suggest leaving the defaults alone (and unselected) and using the Add button (though you may be sick of Wizards at this point) to select what you want to happen.

Note that this screen allows you to either Block, Permits, or Negotiate security when a filter rule if encountered. I will choose Negotiate Security, and click the Add button, and choose that High security (ESP and AH) should be used.

You can also configure custom settings, if you require them. Be sure to give your filter actions a descriptive name (and select it).

The next screen simply finishes the process. Remember that the policy is still not assigned – right click the policy and choose Assign to do this.

Also remember that IPSec configuration is not a one-way street. You will also need to configure the appropriate settings on your servers to ensure that the policies can negotiate security and function correctly.

A couple of additional notes on IPSec:

– in order to test whether IPSec is working properly, using the IPSECMON utility. It will show you any security negotiations and sessions currently in place.

– Note that you can use Services in Computer Management to stop and restart the IPSec Policy Agent if necessary. This is also the equivalent of stopping and reloading the IPSec driver.

Author: Dan DiNicolo

Dan DiNicolo is a freelance author, consultant, trainer, and the managing editor of He is the author of the CCNA Study Guide found on this site, as well as many books including the PC Magazine titles Windows XP Security Solutions and Windows Vista Security Solutions. Click here to contact Dan.