First (OH)NUG Virtual Event

Due to the restrictions imposed by the COVID-19 pandemic, we were forced to change the Ohio Networking User Group aka (OH)NUG meeting to be a virtual meeting that we hosted on Zoom July 31st, 2020. Despite this, we carried on (with beer, of course) and for the last time presented the great debate between the Layer 2 Leaf Spine (L2LS) vs Ethernet Virtual Private Network (EVPN) datacenter architectures. As always, Chris Kane and Mitch Vaughan had a lot of fun with it and presented their respective “platform”. We put it to the (OH)NUG audience with a poll and it was close but the winner: Layer 2 Leaf Spine

We took a brief break to rearrange the room (and get more beer) for a panel discussion with Bill Blake, Phil Davis and Paul Browning to talk about network automation, the impact of cloud infrastructures on the network and how COVID-19 has changed things for the average network engineer. Great discussion with many insights!

For those that couldn’t attend, keep your eyes peeled for a recording of the event to be posted to the (OH)NUG site soon. Once it’s out, I’ll be sure to post it here as well.

To learn more about (OH)NUG visit or the MeetUp page.

Here’s hoping we see you at the next event, be it virtual or physical!

VXLAN + EVPN in Campus Switching?

I recently received some Arista 720XP PoE Campus Switches a key component of Arista’s Cognitive Campus efforts. While reviewing the datasheet, noticed something I found interesting: VXLAN + EVPN support.

Detail of VXLAN + EVPN support from Arista 720XP Datasheet

For those unfamiliar with VXLAN & EVPN here are some resources to check out.

TLDR; VXLAN is layer 2 Ethernet transport over a layer 3 tunneling encapsulation while EVPN is a BGP based control plane for layer 2 and layer 3 hosts to signal this transport.

It is interesting to have these features in a campus switching platform and got me thinking about the evolution of the campus. Many large scale campus environments that I have worked with over the years have a mix of Layer 2 and Layer 3 islands which can be complex and difficult to deploy, manage and integrate. VXLAN + EVPN can potentially make operating a large, complex network much easier by giving one the capability to build an Layer 3 core that can use VXLAN as a transport for Layer 2 across a campus. Couple with that with maturing network controllers and intent based systems then the potential is there to build seamless, scalable infrastructures. For some of the huge, technology debt laden campuses out there, relief may be coming to reduce the difficulties of managing these systems.

Though use of VXLAN + EVPN in the campus is not new with vendors like Cisco, Extreme and Juniper offering solutions in the past few years as well, uptake seems to be slow. I often find many network engineers are rarely familiar with VXLAN much less EVPN so I think it’s going to be a while before they really get traction. I also think the software management platforms will be key to obfuscate the complexity and it will take a while for those to mature as well.

I like where this is headed and can see a lot of value in it for network practitioners. That said, one should always weigh the trade offs and validate that the complexity is warranted. Introducing technologies like EVPN to an environment is not trivial and it is key to understand how to operationally handle it including ongoing management and troubleshooting of these technologies. As always, be sure to lab and really dig into how things work before deploying.

(OH)NUG Fall Event 2019

What a fun event this past Ohio Networking User Group meeting was in Cleveland! I joined 120 other Ohio networkers at the Brew Garden in Strongsville for what turned out to be an extremely fun and insightful event. The (OH)NUG membership and events have grown substantially since our first one in February 2017 which had a grand total of 18 attendees. Here’s to a lot more growth and thought provoking conversations in the days to come!

The event started with a debate between Layer 2 Leaf Spine (L2LS) vs Ethernet Virtual Private Network (EPVN). Fellow (OH)NUG founders Mitch Vaughan (team EVPN) debated Chris Kane (team L2LS) the merits of their respective technologies. I was fortunate enough to moderate this debate which certainly was lively. These guys really played the role of a candidate ferociously making the case for their platform and really gave each other some hard hitting (and hilarious) jabs! These guys are buds in real life who understand that each technology has its place in the right environment so concluded the debate with a hug followed by a town hall style discussion breaking down where you would use each.

We took a little break then closed the event with an expert panel which included Bill Blake, Tom Zukowski, Brandon Greer and Luke Teeters expertly moderated by Mitch Vaughan. Topics covered were whether network engineers are really seeing orchestration and automation making an impact in their daily lives, the impact of cloud to network engineers and security challenges netengs encounter. So many insights came out of this incredibly interactive conversation with the audience engaged and asking some fantastic questions.

I’m really proud of what this little group has become. It really is a safe space where network engineers can get together to talk networking in a sales and vendor free environment… with beer. Just as we intended. Hope to see you at the next one!

Maintenance Windows: Keep it short or use it all?

So I heard about an interesting encounter which started a debate with our team. An engineer on our team was working through a maintenance window and was prepared for the event with all of his configs and the order of operation to complete the maintenance as quickly as possible. This was all shared with the customer. Even though there was an hour set aside for the window with change control, our engineer wanted to use as little of that time as possible for downtime and stick to the change as specified, making the impact to users minimal. The engineers on the customer end said “We have an hour, there’s no rush.” so took their time moving cables without any prior preparation or planning to do so quickly. The customer also seemingly had no urgency to make the configuration changes quickly to keep downtime to a minimum, assuming they had plenty of time. This experience started a conversation about this and our team came to the conclusion that this perspective was pretty common.

There is something to be said for being methodical and carefully working through the maintenance. That said, I would contend that a sense of urgency to complete the maintenance quickly reduces impact to users and gives more time to troubleshoot if any issues arise. I personally make it a point to get the maintenance done quickly with as little downtime as possible but it seems that some choose to use the all of the allotted time.

If it was your maintenance window to run with, how would you handle it? Get it done as quickly as possible with as little downtime as you can or take your time and use the whole time window available?

Making the Business Case for SD-WAN

There are many things to consider for businesses to adopt a new technology into their environment. SD-WAN has emerged as a better way to build the WAN for organizations as it improves performance, adds greater redundancy and many times can reduce cost. I was fortunate to work with the Packet Pushers to create a white paper on making the Business Case for SD-WAN. Full disclosure, you do need a Packet Pushers Ignition account which is a bargain at a mere $99/yr for resources like this and access to their Slack channel in which you can find some pretty awesome networking conversations! If you want to check out the mini version for free, WAN Dynamics has a brief posted on our site here:

I hope you find value in it and if you need to follow up, reach out to me on Twitter or LinkedIn!

Connecting the Cloud: SDN, SD-WAN and Multicloud

On the tail of a new report that details how SDN & SD-WAN are becoming a mainstream consideration within many organizations, we have been pondering on why SD-WAN and cloud based network solutions have struck such a chord. 2017 was an incredible year of transformation in the network industry and 2018 is shaping up to be even bigger. The following are some thoughts on what is going on with SDN, SD-WAN and the modern cloud connected network this year.

Last year saw a massive uptick in public cloud adoption within companies and the technological disruption of many traditional, established businesses. We see clients every day overcome by regulatory compliance considerations, security concerns, stacked up operational and technical debt not to mention countless other complex IT challenges. This is forcing many to rapidly move out of traditional models of maintaining their own private IT infrastructure and applications internally to finding cloud based alternatives in order to gain efficiency needed to keep up. Organizations are more frequently than ever looking at their business problems through the lens of the cloud and are beginning to understand the promise and accompanying value, finally prepared to accept change. From where we sit, adoption is widespread and it’s across the board in every vertical and every size of customer. What we see time and again is that as attempts are made to transition to cloud services, legacy infrastructure pieces which are key to supporting cloud efforts like connectivity and security are neglected. With comprehensively managed network services like those that WAN Dynamics provides (SD-WAN, firewall, etc), we were able to assist many to get back on track to accomplish what started their cloud ambitions in the first place: Providing greater agility and value to their core business by enabling cloud adoption. We are firm believers in the cloud but in order to host services there, the infrastructure needs to be ready for it. It was a great year of discovery, education and growth along side our clients helping them solve so many of these new challenges as we go.

The pent up demand to solve problems of leveraging hybrid WAN connectivity options coupled with SD-WAN has been akin to a tidal wave. Most businesses are facing the same issues around intelligently using available connectivity in an active and dynamic capacity, addressing bandwidth quality problems over paths on the fly, seamless failover and truly uniform, holistic visibility and management of the network. Customers we meet with have heard of SD-WAN, understand the key value proposition and know that it is something requiring further exploration. As it becomes mainstream and starts to replace older WANs, we seeing a bit of backlash from the legacy vendors and service providers unable to adapt or find ways to replace shrinking revenue streams from selling and supporting legacy WANs. SD-WAN works in conjunction with a multitude of connectivity options, even L3 VPN/MPLS so for those that wish to hang onto their traditional L3VPN WAN to use with SD-WAN, they can. That said, we find most SD-WAN deployments will take advantage of the direct connectivity over Dedicated Internet Access (DIA) and Broadband and will displace MPLS connections. This will aid in better management of cloud connectivity for the organization with local Internet breakout at locations which require it.

Also of significance is the emerging trends around managing unified network policy within datacenter, cloud and WAN. Networking vendors across the board are working out “Multi-Cloud” strategies to cohesively stitch together all of these environments. Doing so will be key to scale as fragmentation of point solutions become exacerbated by maintaining many different types of infrastructure.

Let’s also not discount security’s role. With so many emerging threats coming at us daily and regulations such as General Data Protection Regulation (GDPR), security is top of mind for many in IT so watch for the cloud and premises based security integrations with the network to mount. There will be many new offerings deployed with “service chaining” which can make tie network elements together in different places be it onsite, potentially as a virtual network function (VNF) or virtual machine running on a general purpose x86 network appliance or even tunneled to a cloud service, making things dynamic and scalable. Most security offerings are pivoting to accommodate cloud, so if a company has a centralized or already well defined security posture which they would like to maintain, vendors are making it easier to do so and easily integrate.

2018 will mark a year of growth in the networking industry that has not been seen in some time. Rapid mainstream adoption of SD-WAN and cloud connectivity options will continue and it will become a core element for network designs from here on out. Those exploring a network refresh without considering the impact of the cloud are doing themselves and their business a disservice. We are looking forward to all of the new applications and opportunities we expect to be a part of!

How Software Defined Wide Area Networking (SD-WAN) Provides Reliable Voice and Video Services Over the Internet

For as long as organizations have tried to make real-time services like voice or video work over Internet Protocol (IP) network pipes, there have been very basic requirements in order to make said services operate effectively. The first requirement for these sensitive applications was a dedicated, business class network line to carry this traffic. A business class circuit was paramount to reliability and uptime required for crucial services like voice or video. This type of network access has low latency characteristics which keeps the amount of time it takes to forward the voice traffic low so that conversations are not made off kilter by long delays.

Also absolutely critical to voice or video over network pipes is an additional layer over these high quality dedicated connections, something called quality of service or QoS. QoS is a suite of bandwidth prioritization and reservation techniques that give select services fast lane access to bypass lesser classifications of traffic and also reserves bandwidth preventing exhaustion of available throughput. Most commonly, QoS is used in tandem with carrier services like an IP VPN or Multi-Protocol Label Switching (MPLS) and have been assumed by many to be the only way to reliably deliver voice services for an organization. I can affirm as a network engineer for the past few decades, this has been the case for most of my career. In order for voice to perform adequately, specific care was required to specify dedicated pipes with prioritization and if you did not perform technical due diligence, you were asking for trouble in the way of poor quality, session disconnections and general voice issues.

Then something called Software Defined Wide Area Networks or SD-WAN came along. This nascent technology space is drastically changing the way we do a lot of things on the wide area network, including managing sensitive real-time protocols that typically require QoS. Read more on what SD-WAN is here.

Let’s take a look at some of the mechanisms that make SD-WAN different versus how we’ve implemented voice over traditional networks up until now. Though many of these techniques may not qualify specifically as QoS, they mimic the capabilities and allow for more reliable Internet based infrastructure to support real-time protocols. The combination of these techniques that have been used individually for decades, create a service greater than the sum of its parts. Features now considered fundamental aspects of most SD-WAN platforms are differentiators from the means we have used in the past to run network traffic over networks.

  1. Multi-Path Steering – SD-WAN can actively forward over multiple paths and is constantly measuring the performance characteristics and properties of each path available. Because it can very rapidly identify issues like high latency, packet loss and jitter, there are software mechanisms to quickly bypass these issues by utilizing an alternate, better performing path on the fly.
  2. Forward Error Correction/Packet Duplication – When issues like data loss from dropped packets arise, if there is only one path available or all paths are experiencing loss, that can be a serious issue with traditional networks with little means to remediate. SD-WAN employs features such as Forward Error Correction (FEC) or packet duplication, which becomes enabled once packet loss is identified on a path.  This technique will send duplicates of each packet in the flow over a single path or over multiple paths to have greater assurance that critical data like voice or video will make it to the destination. At the other side of the session for that voice or video stream, the first packet received will be forwarded to the destination and the duplicates packets will be dropped but if packets are dropped, the duplicated packet will be used in its place.
  3. Jitter Buffering – Voice and video quality can suffer from a network condition called “jitter” which is when the information sent over the network is spaced inconsistently leading to a variable tempo for the stream. The result is audio or video that can have gaps in timing and become impaired. SD-WAN measures the gaps between the packets and can evenly space these packets on the other side providing what is called a “jitter buffer” to realign the timing of these packets to keep the video or audio stream cadence intact.  Jitter buffering has been performed before but traditionally at the application servers and endpoints (i.e. IP phones or IP video appliances).  The unique differentiator for SD-WAN  is performing this inline on the network versus relying on the end points and application servers to supply the jitter buffering.
  4. Prioritization and Queuing over Multiple Tunneled Paths – Because SD-WAN performs it’s queuing and packet forwarding over something called an “overlay”, the forwarding decisions for information that has the highest priority and reservation of bandwidth for applications is performed at a layer above the traditional IP interface. With this, a priority “fast pass” can be given to crucial data like voice, video or other business essential apps bi-directionally and this can be done over all paths available. These overlays are typically facilitated with tunnels over top of existing infrastructure versus on the actual underlay interfaces.  This allows user defined packet queuing and service prioritization configuration overtop of service provider links.

So as you can see, there are many pieces that come together to make IP based voice over broadband and Dedicated Internet Access (DIA) now possible. Our organization has played a part in designing many SD-WAN based solutions for customers and have seen it perform in the “real world” so can attest first hand, it works.  We are beginning a new era of intelligent, self-healing networks which Software Defined Networking (SDN) applications like SD-WAN will be leveraged to usher in.  Though many of the technologies leveraged by SD-WAN are not new, the way they are put together and managed by an SDN controller is and it is this combination that makes it truly powerful. It is with great confidence that I can state, SD-WAN is not a fad and it will be a fundamental piece of how organizations will build out their connectivity moving forward.

Five Use Cases for SD-WAN

A lot of folks I speak with about Software Defined Wide Area Networking (SD-WAN) are trying hard to understand how this rapidly emerging technology works and the places where it can fit with their clients or within their own network. As we acquire more experience with deployments inside many different business and network environments, the results that we discover are quite surprising. There are many applications where SD-WAN is an obvious fit but in some cases, the true value is not exactly what we were expecting. The following are some of the more prominent examples of reasons for SD-WAN we’ve been able to assist with to date:

  1. Voice Services Over the Internet – A lot of small to medium sized businesses have started utilizing voice services over commodity broadband connections with no Quality of Service (QoS) in place. Though most of the time this works adequately, there will be many instances of degradation in quality or dropped calls that can be frustrating. This has just been the reality of utilizing the public Internet for voice services… up until now. With SD-WAN, we’re able to prioritize voice traffic both inbound and outbound while leveraging multi-path technologies to “route around” carrier backbone problems. We’re able to do this with single, stand alone sites in addition to multiple locations.
  2. WAN Visibility and Management – Setting aside the benefits of multi-path link steering, bandwidth aggregation and QoS for a bit, many organizations have no usage breakdowns or application performance visibility in their network today. As a byproduct of the application steering and prioritization baked into most SD-WAN solutions, there is a great deal of reporting functionality available. So now when stakeholders of IT want to know what is happening at their remote locations, they have a graphical interface to see exactly what is happening.
  3. Configuration Uniformity and Standardization – Large organizations which have many sites or will soon have many sites at the hands of rapid growth can have a lot of hands in the IT group working on things. With this, lack of standardization becomes an issue as sites are configured and turned up if there is not a uniform configuration policy. With SD-WAN, attaining a high level of uniformity is simple using features like Zero Touch provisioning and Configuration Profiles to make sure that all sites are configured identically. This also helps greatly for change management if you want to make a configuration update to all of your locations. With this approach, you can update a configuration in one place and push it to all sites, instantaneously. This frees up engineers to solve larger problems facing the business rather than making a minor configuration change on dozens or hundreds of sites.
  4. Remote Diagnostics Capabilities – When there are issues at a remote location, it can often times be difficult to walk users through providing troubleshooting assistance or getting the right software and hardware onsite. With the built in tools into many SD-WAN solutions, the ability to perform packet captures, see network state and what the users see on the network, so that the time vetting issues on the network can be greatly reduced.
  5. MPLS / IP VPN Replacement – MPLS and other dedicated private network infrastructures have begun to outlive their usefulness with many organizations as critical workloads are moved to the cloud. Further, there is growing demand by companies to reduce cost of their expensive WANs that typically have no redundancy or application smarts built in. SD-WAN can easily leverage existing dedicated internet access (DIA) links and even inexpensive broadband connections to build an application aware, private network overlay that provides more applications control, redundancy and critical business application prioritization than traditional network designs.

These are just five examples of things we have been able to help with. We’re happily conducting Proof of Concept deployments for businesses to show the value of SD-WAN and finding new use cases all the time. We find ourselves working on long standing problems that have been occurring for years in traditional networks and within just a few hours of having SD-WAN appliances in the network, fixing them. Using this technology is some of the most rewarding work I’ve ever done as a network engineer. SD-WAN really is a game changer!

Should your telecom provider manage your SD-WAN strategy?

As interest in software defined wide area networking (SD-WAN) grows, many traditional telecommunications providers are jumping on the bandwagon to bundle their data and voice service offerings.  I would caution those exploring their options with SD-WAN as a potential technology solution for their business to think carefully about going this route.  I’ll detail some key reasons why one may want to steer clear of bundled offerings from telecommunications carriers:

  1. A key benefit of SD-WAN is the freedom from carrier lock in and the ability to select the best access provider(s) a particular region has to offer. Bundling with your telecom provider can hamper flexibility to pick and choose the circuits you want.
  2. When bundling with circuits, pricing may prove unpredictable when the packaged offering is pulled apart.  If it’s determined later to choose another SD-WAN solution, to take circuits to other providers or to change the arrangement in any way, it could adversely affect the bottom line and there may even be penalties.
  3. Carriers will represent one, maybe two SD-WAN solutions at most with which vendor partnerships are secured.  Because there is no “one size fits all” model with any vendor currently in the SD-WAN space, Managed Services Providers (MSPs) and Value Added Resellers (VARs) have a compelling story with more choice and a better technological fit as they can represent many different best of breed solutions.
  4. A Managed Service Provider offering provides more of a customized and “boutique” solution which can be tailored to the customer needs.  Service provider offerings are typically standardized and very rigid leaving little room to get “out of the box” to provide advanced integration options.
  5. Telecommunications carriers have a vested interest in maintaining the high margins of services like MPLS.  With that, SD-WAN service offerings will likely be created to augment MPLS services, not replace them whether or not that is the right solution for the customer.

From a technology standpoint, SD-WAN will no doubt create a great deal of value, agility and savings for those running large wide area networks, no matter who it is procured from.  That said, I would advise not locking into a solution that reduces choice and the ability to realize SD-WAN’s fullest potential.

Why I left my job to co-found a managed SD-WAN services company

I left my job of 13 years to found a new company.  It’s the end of an era for me and it was not an easy decision to make.   The reason?  I believe a new technology space known as Software Defined Wide Area Networking (SD-WAN) is one of the most significant advancements we’ve seen in decades as to how we build wide area networks and I want to be a part of it.  I’ll explain.

I’ve worked for one service provider or another for the past 17 years.  I joined Stratos Internet Group, a regional dial-up ISP, on the same day that Naptser debuted on June 1st, 1999.  From dial-up to Carrier Ethernet, frame relay to MPLS, e-mail and website hosting to cloud, there has never been a shortage of new services emerging and exciting technologies to learn about.  In particular, since 2003 I’ve enjoyed helping build a regional business class ISP called Fidelity Access Networks into Fidelity Voice and Data, a boutique telecommunications powerhouse in Ohio.  We married a carrier grade network with out-of-the-box thinking and an unparalleled support experience to offer what I feel were the best products in the business.  Fidelity was a large and very important part of my life for the last 13 years.   As many of you who have worked with us know, Fidelity was acquired by Fusion (FSNN on Nasdaq – back in December of 2015.  Fusion is a good fit from a product and services perspective plus fills in the gaps with many services that Fidelity lacked.  Even though all of this was lining up to be an interesting next chapter for me, there was something on my mind.  I felt myself drawn in a different direction.

Enter my interest in this thing called Software Defined Wide Area Networking or SD-WAN.  If you’re in the field of telecommunications or in IT and this is the first you’ve heard of SD-WAN, I assure you, it will not be the last.  I first discovered it in 2012 while I was working on an MPLS deal with an agent partner and one of our sales folks at Fidelity.  We were up against a company that was a very early entrant in the SD-WAN market.  I’d never come across anything like it before but after learning more about their solution, I was impressed.  To be honest, after understanding what they were doing, I found our MPLS solution inferior.   The SD-WAN service was able to tunnel private traffic over commodity public broadband links, aggregate the total throughput of available links so there was never one sitting idle, it achieved redundancy across the connections without a complicated dynamic routing protocol, provided centralized policy and control with application based QoS for a fraction of the price of our offering.  That was just incredible to me.  What a fascinating concept and an absolutely disruptive technology.

After giving all of this more thought, I came to a realization.  Though some minor features and functionality have changed over the years, we have fundamentally been building the same networks since my career began in 1999.  Static, complex and closed.  That said, the advent of cloud services is changing the way we work and is forcing the network to change with it.  Business critical services like ERP and CRM applications, unified communications, hosted voice and video conferencing are moving off of the traditional private corporate network and onto the public, virtual infrastructure we call “the cloud”.   Added to that, the economics of commodity broadband (Cable Internet, DSL, 3G/4G, etc) are beginning to overshadow the value of the symmetrical bandwidth, SLAs and perceived reliability of dedicated links.  The proposition of finding smarter ways to use these commodity connections to provide reliable, high performance connectivity to cloud resources is an undeniable driver so will no doubt continue.  But how do you make these services work over the public internet without tools like QoS or the visibility and control of dedicated links?  It is my belief these objectives will be achieved with SD-WAN.  This new approach to networking will give organizations the ability to put their mission critical apps on commodity cable modem or DSL services to realize performance expectations traditionally found on services like MPLS or dedicated lines.   That is why we at WAN Dynamics will be helping organizations build value driven SD-WAN networks from now on.  There’s no doubt in my mind that in 5 years, organizations who are NOT managing their sites with SD-WAN will be the outliers.