Maintenance Windows: Keep it short or use it all?

So I heard about an interesting encounter which started a debate with our team. An engineer on our team was working through a maintenance window and was prepared for the event with all of his configs and the order of operation to complete the maintenance as quickly as possible. This was all shared with the customer. Even though there was an hour set aside for the window with change control, our engineer wanted to use as little of that time as possible for downtime and stick to the change as specified, making the impact to users minimal. The engineers on the customer end said “We have an hour, there’s no rush.” so took their time moving cables without any prior preparation or planning to do so quickly. The customer also seemingly had no urgency to make the configuration changes quickly to keep downtime to a minimum, assuming they had plenty of time. This experience started a conversation about this and our team came to the conclusion that this perspective was pretty common.

There is something to be said for being methodical and carefully working through the maintenance. That said, I would contend that a sense of urgency to complete the maintenance quickly reduces impact to users and gives more time to troubleshoot if any issues arise. I personally make it a point to get the maintenance done quickly with as little downtime as possible but it seems that some choose to use the all of the allotted time.

If it was your maintenance window to run with, how would you handle it? Get it done as quickly as possible with as little downtime as you can or take your time and use the whole time window available?

Five Use Cases for SD-WAN

A lot of folks I speak with about Software Defined Wide Area Networking (SD-WAN) are trying hard to understand how this rapidly emerging technology works and the places where it can fit with their clients or within their own network. As we acquire more experience with deployments inside many different business and network environments, the results that we discover are quite surprising. There are many applications where SD-WAN is an obvious fit but in some cases, the true value is not exactly what we were expecting. The following are some of the more prominent examples of reasons for SD-WAN we’ve been able to assist with to date:

  1. Voice Services Over the Internet – A lot of small to medium sized businesses have started utilizing voice services over commodity broadband connections with no Quality of Service (QoS) in place. Though most of the time this works adequately, there will be many instances of degradation in quality or dropped calls that can be frustrating. This has just been the reality of utilizing the public Internet for voice services… up until now. With SD-WAN, we’re able to prioritize voice traffic both inbound and outbound while leveraging multi-path technologies to “route around” carrier backbone problems. We’re able to do this with single, stand alone sites in addition to multiple locations.
  2. WAN Visibility and Management – Setting aside the benefits of multi-path link steering, bandwidth aggregation and QoS for a bit, many organizations have no usage breakdowns or application performance visibility in their network today. As a byproduct of the application steering and prioritization baked into most SD-WAN solutions, there is a great deal of reporting functionality available. So now when stakeholders of IT want to know what is happening at their remote locations, they have a graphical interface to see exactly what is happening.
  3. Configuration Uniformity and Standardization – Large organizations which have many sites or will soon have many sites at the hands of rapid growth can have a lot of hands in the IT group working on things. With this, lack of standardization becomes an issue as sites are configured and turned up if there is not a uniform configuration policy. With SD-WAN, attaining a high level of uniformity is simple using features like Zero Touch provisioning and Configuration Profiles to make sure that all sites are configured identically. This also helps greatly for change management if you want to make a configuration update to all of your locations. With this approach, you can update a configuration in one place and push it to all sites, instantaneously. This frees up engineers to solve larger problems facing the business rather than making a minor configuration change on dozens or hundreds of sites.
  4. Remote Diagnostics Capabilities – When there are issues at a remote location, it can often times be difficult to walk users through providing troubleshooting assistance or getting the right software and hardware onsite. With the built in tools into many SD-WAN solutions, the ability to perform packet captures, see network state and what the users see on the network, so that the time vetting issues on the network can be greatly reduced.
  5. MPLS / IP VPN Replacement – MPLS and other dedicated private network infrastructures have begun to outlive their usefulness with many organizations as critical workloads are moved to the cloud. Further, there is growing demand by companies to reduce cost of their expensive WANs that typically have no redundancy or application smarts built in. SD-WAN can easily leverage existing dedicated internet access (DIA) links and even inexpensive broadband connections to build an application aware, private network overlay that provides more applications control, redundancy and critical business application prioritization than traditional network designs.

These are just five examples of things we have been able to help with. We’re happily conducting Proof of Concept deployments for businesses to show the value of SD-WAN and finding new use cases all the time. We find ourselves working on long standing problems that have been occurring for years in traditional networks and within just a few hours of having SD-WAN appliances in the network, fixing them. Using this technology is some of the most rewarding work I’ve ever done as a network engineer. SD-WAN really is a game changer!

The Service Provider IGP Question: OSPF or Integrated IS-IS?

(Moved from my old blog, http://packetrancher.com, which I decided I didn’t have the time for so shuttered in 2011. This was one of the few blogs posts worth saving from it.)

I had a choice to make recently in the decision of which open standards based IGP Routing protocol (i.e. NOT EIGRP) to chose between, OSPF or Integrated IS-IS.  If you look out there on the Internets, you’ll find many, many different discussions about which one to go with.  There are a lot of engineers who think IS-IS is dead and that no one uses it anymore, often times confusing it with IGRP (which SHOULDN’T be used anymore).  That is far from the truth as most large networks have used IS-IS for years and many others switch to it all the time.

There are positives and negatives to both OSPF and IS-IS as you’d expect, but they are very similar protocols.  First, lets get a run down of some of the facets and features of each:

OSPF

  • Version 1 became RFC 1131 in October 1989
  • Uses Dijkstra’s Algorithm to determine shortest path
  • Distributes routing updates/information with LSA (Link State Advertisement)
  • Runs over Internet Protocol (IP)
  • Supports Non-Broadcast Multi-Access Networks (NBMA) and Point to Multi-Point (P2MP) in addition to Point to Point (P2P) and Broadcast
  • Partitioned into ‘Areas’ where Area 0 is the backbone that connects all other areas.
  • IPv6 support: Added with re-written version 3 of the protocol

Integrated IS-IS

  • Published as RFC 1195 in December 1990
  • Uses Dijkstra’s Algorithm to determine shortest path
  • Distributes routing updates/information with LSP (Link State Packet)
  • Runs over ConnectionLess Network Protocol (CLNP)
  • Unnumbered Broadcast in addition to Point to Point (P2P) and Broadcast. No NBMA or P2MP
  • Possible to be partitioned into ‘Levels’ where Level 2 is the backbone that interconnects all other Level 1 areas
  • IPv6 support:  Was added with a Type-Length-Value (TLV) addition to the protocol

As you can see, a lot of similarities.  In fact, when most network engineers who have experience in both are asked which they would recommend, they say it really comes down to preference because they are so similar.  Which protocol are your engineers accustomed to using and troubleshooting with?  That’s the one to go with.  I think it’s a little more involved than that, but from an network operations perspective I guess that could be a determining factor.

In evaluating my network to see which is going to be the best long term fit, I’ve come to the conclusion that Integrated IS-IS is the right choice for me.  There are a number of reasons why I came to this conclusion.

  1. Security – IS-IS runs in CLNP, not IP.  This means it is not as vulnerable to IP spoofing or other denial of service attacks that could affect OSPF.  Also if you run MPLS VPNs with OSPF in them, you’re less likely to have a NOC engineer accidentally add a customer to your core OSPF topology.
  2. Modularity – Equipment vendors can easily add newer protocols or features into IS-IS with the addition of new TLVs and sub-TLVs.  OSPF has historically required a re-write from the ground up to add support for protocols such as IPv6.
  3. Reputation – There is a very high opinion of IS-IS within engineering circles as being rock solid, quick converging and a very predictable IGP.  Granted, this is hearsay from my colleagues at other service providers, but I consider their opinion very valid.
  4. Simplification – I like the idea of keeping things simple so running IS-IS as both my IPv4 and IPv6 IGP is attractive.  In an OSPF world, that would require two routing instances, one for OSPFv2 routing IPv4 and the other for OSPFv3 routing IPv6.  I also think OSPF has too many knobs to play with that can let operators get a little carried away to make their networks more complicated than necessary.
  5. Vendor Focus – IS-IS is used predominantly and almost exclusively in the service provider space.  This creates a laser like focus of features and development on what service providers need.

So am I saying Integrated IS-IS is the best interior routing protocol ever invented that everyone should use?  By no means.  As with most comparisons of technologies so close to each other in operation, it comes down to the application of the technology.  Make sure you dig into the subject matter to get a good understanding so that you can really make a business case for your solution.  In decisions like the choice of an IGP, it’s something you are likely going to be stuck with for some time.  To swap it out for another protocol can be an absolute bitch to plan, test and change especially as the network grows.  It’s best to build it once so that it is stable and scales in YOUR environment.

Here’s a few great resources on the subject of ISIS vs. OSPF if you’re interested to read more: