Transcript Matt Baker Presentation

Very Pleasant/Painful Networking:
The Highs and Lows of Building and Maintaining
IPsec Based Customer Access VPNs
Matthew W. Baker
Intel Online Services, Inc.
NANOG22
VPN - Behind the Scenes
IPsec based VPNs are highly effective, and
can prove to be very valuable for a myriad of
applications. However, building and
maintaining Virtual Private Networks can be
difficult and frustrating. This presentation
will highlight some issues that frequently
pop up, and some strategies for dealing with
them.
Intel Online Services Confidential - 2
Agenda
Introduction/Background
The Distilled Taxonomy of IPSec VPNs
Technology Challenges
Bad NAT
Troubleshooting Complexity
The Many Levels of Interoperability
Environmental Challenges
Quality, Reliability, and Performance
External Security Stumbling Blocks
Intel Online Services Confidential - 3
More Agenda
Environmental Challenges
Security Concerns
Routing and Addressing
Good NAT
Human Factors
Conclusions
Intel Online Services Confidential - 4
Distilled VPN Taxonomy
Site-to-Site/LAN-to-LAN/Branch Office
VPN connecting two networks, or groups of
networks.
Typically employ main mode IKE with preshared keys or certificates for authentication.
CPE based device-to-device...normally either an
edge or edge-1 gateway device.
Routing and addressing management is a
factor
Security of tunnel relies on integrity of the
participating networks.
Intel Online Services Confidential - 5
LAN-to-LAN VPNs
Intel Online Services Confidential - 6
Distilled VPN Taxonomy
Client-to-LAN, Remote Access
A VPN tunnel connecting a single node to a
remote network.
Typically employs aggressive mode IKE with
pre-shared keys/passwords, certificates,
tokens, etc. for authentication.
Client software driven
Security appears more tightly controlled
Access policies can be centrally managed
Intel Online Services Confidential - 7
Client-to-LAN VPNs
Intel Online Services Confidential - 8
Technology Challenges
Bad NAT
Network Address Translation presents many
difficult challenges.
IPsec has inherent issues with NAT and vice versa,
“many-to-one” NAT is particularly problematic.
Knowing how any single NAT
implementation will effect IPsec is
impossible...Assume the worst!
Some NAT implementations completely kill IPsec.
Others will allow a single tunnel to be created which will
be killed by subsequent attempts to create additional
tunnels
Intel Online Services Confidential - 10
Bad NAT
The use of NAT is pervasive in the
broadband and low cost access markets.
Large LECs utilize broadband CPE based NAT to
ease implementation complexity and conserve IP
space.
Many end users are usually unaware of the
nature of their Internet connectivity.
Realm Specific IP (RSIP) solutions slow on
the uptake.
Intel Online Services Confidential - 11
Strategies: Dealing with Bad NAT
Be prepared with customer documentation
describing how to create 1:1 NAT between nodes
requiring VPN access and the NAT devices.
Be prepared to assist customers by facilitating
communication between customer and their service
provider.
If customers will require access from many nodes
behind a NAT gateway, consider LAN2LAN access.
Consider another VPN implementation that will allow
NAT traversal? Is a UDP wrapped client available?
Leverage your vendor!
Intel Online Services Confidential - 12
Troubleshooting Complexity
Troubleshooting toolsets remain fairly
immature.
Varying “Standard” implementations make
root cause fingerprinting difficult.
The very nature of Virtual Private Networking
makes troubleshooting extremely difficult.
Sniffing of packets is essentially useless, packets
are homogenous and encrypted.
Intel Online Services Confidential - 13
Strategies: Troubleshooting
Build an IPsec debug target with strong
logging capabilities.
Reduce complexity by enforcing product
standards, and ensuring number of
device/vendor combinations is minimal.
Structure access network with a device that
accommodates simple packet inspection.
Intel Online Services Confidential - 14
The Many Levels of
Interoperability
Cross Vendor Interoperability
Varied IPsec implementations make cross vendor
interoperability troublesome at best
Many critical features are often vendor specific.
Products are often streamlined for usability, thus
protocol extensions and other proprietary
features cannot be disabled.
C2L interop is particularly troublesome. VPN
clients have become commodities, thus the goal
of interoperability rare.
Vendor support is problematic
Potential cost savings are often consumed by
implementation minutia and unpredictable
stability/performance.
Intel Online Services Confidential - 15
The Many Levels of
Interoperability
Intra Vendor Interoperability
Typically vendor sanctioned and supported
Operationally difficult given disparate command
interfaces, orders of operation, etc...
Client-to-LAN – OS/Application Interop
Will a particular client run on all operating
systems???
Unix flavors are conspicuously absent from most
vendor’s OS support lists!
Multiple VPN client installed simultaneously
causes issues
Expect application/client interoperability issues
Intel Online Services Confidential - 16
Strategies: Interop
Enforce standard device/client combinations.
Strongly set expectations and share your
“supported standards” upfront with
customers.
Proactively publish known compatibility
issues at all levels.
Test, test, test!!!
Intel Online Services Confidential - 17
Environmental Challenges
Quality, Reliability, and
Performance
Customers are many times unaware of the
costs associated with using the Internet for
mission critical data transport.
Application issues may arise as increased
latency and bandwidth variance are inevitably
introduced.
Varying performance introduced by the
vagaries of the Internet are difficult to isolate
and difficult to explain.
The cost of QR&P investigations is high.
Intel Online Services Confidential - 19
Quality, Reliability, and
Performance
Customer ISP selection is a factor!
The quality of customer Internet connectivity will
have an impact on the level of service you are
able to deliver...
Expectations must be set solidly, and service
level agreements carefully crafted.
Beware of custom applications, DB activity,
etc...
Intel Online Services Confidential - 20
External Security Stumbling
Blocks
Many IT organizations are unfamiliar with
IPsec!
Customer’s hesitate to open holes for IPsec.
UDP is evil incarnate to most firewall admins,
IKE openings are therefore most problematic!
ESP/AH are little understood – making them
suspect.
Some ISP have been known to arbitrarily
block tunneling protocols...isolation can be
difficult.
Intel Online Services Confidential - 21
Strategies: QR&P and
Security
Draft SLAs that comprehend the likelihood of
performance fluctuations. Protect both yourself and
the customer by setting concrete expectations.
Educate your customers with regard to the
implication of Internet based networking, e.g.
compare and contrast with leased line performance.
Gather metrics on device utilization, available
bandwidth, etc... This allows for data driven
discussions of performance issues.
Gather Internet performance metrics when possible,
and leverage internet communities such as NANOG
to keep abreast of Internet “events”.
Intel Online Services Confidential - 22
VPN Security Concerns
Client-to-LAN
Considered by many to be more secure.
Scope of tunnel is well defined.
Security policies can be centrally controlled and
pushed out to clients.
Strong user level access control features are
available
Traffic bifurcation presents considerable risk
Route through/Trojan attacks are easily mounted
when customers are allowed to fork traffic.
Intel Online Services Confidential - 23
VPN Security Concerns
LAN-to-LAN
Scope of tunnel is amorphous
Security policies and protective measures cannot
be centrally controlled
Rogue access is simple and straight forward if
security of remote network is compromised
A level of user access control is lost
Intel Online Services Confidential - 24
Strategies: Security
Concerns
Enforce “no bifurcation” policy when using
Client-to-LAN...the risks are simply too great
otherwise.
Favor Client-to-LAN technology as first
option.
Strictly scope L2L tunnels!
Educate customers regarding security issues
with L2L tunnels, e.g. publish white papers.
Expect customer security lapses, and
vigilantly protect your business and network
from these lapses.
Intel Online Services Confidential - 25
Routing and Addressing
Challenges
Client-to-LAN
C2L makes routing simple, and addressing issues
moot. However, the limitations of C2L VPNs
quickly introduce the need for a more flexible L2L
approach.
LAN-to-LAN
Routing and addressing issues abound
Intel Online Services Confidential - 26
Routing/Addressing – An
Example
Customer A Networks:
10.3.3.0/24
10.4.4.0/24
10.5.5.0/24
Customer A
The Internet
Dest 10.3.3.34
Customer B
Routing can quickly
become problematic
when customer end
networks collide!
Customer B Networks:
10.1.1.0/24
10.2.2.0/24
10.3.3.0/24
Intel Online Services Confidential - 27
LAN2LAN Routing/Addressing
Use of private addressing is pervasive,
making collision inevitable.
Advanced routing techniques, like Policy
Based Routing etc., commonly are not
feasible/available.
Route sharing is typically undesirable and
complicated.
Intel Online Services Confidential - 28
LAN2LAN
Routing/Addressing
If route sharing is not used, customers must
figure out how to plumb traffic to their VPN
gateway.
Route leaking/external redistribution must be
considered, and could have disastrous
consequences! Particularly in a hosting
environment.
Intel Online Services Confidential - 29
Good NAT
NAT can prove beneficial when coping with
routing and addressing issues.
Creates layers of abstraction, a buffer.
Can simplify internal routing
Can mitigate customer/internal IP address
collision.
Returns control to service provider
Downsides
Complexity
Bi-directional traffic initiation is problematic
Application incompatibility
Intel Online Services Confidential - 30
Good NAT – Datacenter Access
Example
Customer A
Networks:
10.3.3.0/24
Customer A
Translate Network:
10.4.4.0/24
10.5.5.0/24
192.168.1.0/24
Customer A
The Internet
Customer B
Customer B
Networks:
Customer B
Translate Network:
10.1.1.0/24
10.2.2.0/24
192.168.2.0/24
10.3.3.0/24
Intel Online Services Confidential - 31
Datacenter
Network
Strategies: Routing and
Addressing
Provide documentation regarding routing
strategies...cover the customer spectrum from small
to large.
Consider an architecture that protects your critical
services from misconfigured or leaking routes –
critical in a datacenter environment.
Build abstraction layer into your access network
where addressing/routing can be “normalized”.
Utilize NAT to retain control of addressing.
Have policy in place regarding the use of private
addressing...keep these addressing issues in mind
as you design and redesign your network.
Intel Online Services Confidential - 32
Strategies: Routing and
Addressing
Take a building block approach with your
design...
Layering typically increases the flexibility of your
design
Layering compartmentalizes elements of VPN
infrastructure (connectivity, access control, etc.)
Layering also promotes defense in depth which
can mitigate concern with shared environments
Be vigilant when selecting a VPN product,
many are still built with the enterprise in
mind.
Intel Online Services Confidential - 33
Human Factors
Perceptions and
Expectations
Underlying connectivity architecture is
abstracted.
Traditional troubleshooting techniques can lead
customers astray – underlying transport
network is concealed.
Education is extremely important
Issue ownership conflicts are common
The customer relationship must be well
defined. A stable and consistently performing
VPN environment is a highly cooperative effort.
Intel Online Services Confidential - 35
Perceptions and
Expectations
Performance Expectations
Service level agreements are tricky to craft, and
difficult to enforce for both parties.
Blame games can be common, and are costly to
the service provider.
Internal Expectations
Ensure that your sales and marketing folks are
up to speed with the technology (within reason)
and the capabilities of your design. We all know
the consequences if you do not!
Educate yourself...know your customers.
Intel Online Services Confidential - 36
Education is Paramount
Setting Expectations is
Essential
Intel Online Services Confidential - 37
Conclusions
The large majority of operational issues are
not VPN related per se – they are far more
mundane.
VPN does not mitigate traditional WAN
connectivity issues.
Addressing/Routing is still a major
concern
VPN combines all of these traditional WAN
connectivity issues with the vagaries of the
Internet – quickly magnifying complexity!
Intel Online Services Confidential - 38
Conclusions
External/environmental issues effecting
performance are extremely common, thus
expectations must be clearly communicated
and agreed to.
Issues over ownership of configuration,
connectivity, even hardware are common
Know your target customers well...ensure
that your design will be flexible enough to
accommodate the vast majority with without
breeding exceptions.
Education should serve as the foundation of
the technical relationship with your
customer.
Intel Online Services Confidential - 39