Advancing from 10G to 40G or 100G Ethernet not as simple as swapping out a
few switches or line cards
Upgrading you network from 10G Ethernet to 40G and 100G is not as easy as swapping out switches and line cards.
40G/100G Ethernet
Several factors have to be weighed, such as synchronizing switch clocks for the higher-speeds, especially among multivendor equipment; ensuring latency remains at acceptable levels; keeping the network design and architecture optimal for 40/100G; and making sure the existing cabling infrastructure can accommodate the 4x to 10x increase in bandwidth.
One of the caveats that users should be aware of as they migrate from 10G to 40/100G Ethernet is the need to ensure precise clocking synchronization between systems – especially between equipment from different vendors. Imprecise clocking between systems at 40/100G – even at 10G – can increase latency and packet loss.
The latency issue is a bigger problem than most people anticipate, industry experts say. At 10G, especially at high densities, just the smallest difference in the clocks between ports can cause high latency and packet loss. At 40G, it's an order of magnitude more important than it is for 10G.
This is a critical requirement in data centers today because a lot of the newer innovations are meant to address lower latencies.
“Where you’re going to have the biggest challenges will be different latency configurations if RDMA (remote direct memory access) is used,” says Shaun Walsh, Emulex senior vice president of marketing and corporate development. RDMA is a low-latency, high throughput data transfer capability where application memory is exchanged directly to and from network adapters without copying it to operating system buffers.
“You see a lot more in-rack virtual switching, VM-based switching that is very application specific,” Walsh says. “New line cards in new backplane architectures mean different levels of oversubscription. There’ll be generational tweaks, configuration ‘worrying’ that has to occur. The biggest thing (testers) are running into is making sure you get the 40G you are paying for (with regard to) latency issues, hops, and congestion visibility.”
Emulex late last year acquired Endace, a developer of network performance management tools. Demand for the Endace product and the 40G capabilities of Emulex’s XE201 I/O controller are picking up as more data centers and service providers upgrade from 10G to 40G.
Walsh expects 40G Ethernet to be a $700 million market in four to five years, roughly half the time it took 10G Ethernet to reach that mark. Driving it are next-gen blade server mid-plane interfaces and architectures, big data, analytics, video and data over mobile, BYOD and high frequency trading, Walsh says.
Another challenge is readying the cabling infrastructure for 40/100G, experts say. Ensuring the appropriate grade and length of fiber is essential to smooth, seamless operation.
This is a big consideration for users because it could mean re-wiring a significant portion of their physical plant, if not all of it. That could be an expensive and disruptive undertaking.
At the physical layer, 40G Ethernet is essentially 4x10G “lanes.” But 100G Ethernet is 4x25G lanes, which will be disruptive to the 10G and 40G infrastructure.
“100G is going to be even more of a challenge because now you’re dealing with a whole new layer of physical infrastructure,” Walsh says. “You will have a whole new generation of optics, cables, everything will be a whole new generation at that point.”
Moving bits four to 10 times faster error free is a challenge in and of itself. Making sure the higher level systems – routers and switches – deliver services and non-disruptive service quality at those speeds is equally as challenging, if not more so.
Each device has to do this at one-fourth or one-tenth the time it does at 10G. For a router, it means performing all of the packet inspection, queuing, lookups, filtering, policing, prioritization, table updating and logging while meeting SLAs by not dropping or reordering packets, or increasing latency or jitter.
“Routers aren’t just forwarding packets,” says Scott Register, senior director of product management for Ixia, a maker of test, measurement and visibility tools. “There’s carrier grade NAT, application ID, and security processing and things like that. One of the more interesting testing is, what services can you enable at that rate before you start having problems?”
With carrier grade NAT, the problems get harder as that traffic load increases, Register says. In addition to throughput, increased session capacity and more concurrent connections are also issues as bandwidth climbs from 10G to 40/100G.
“You don’t get more TCP or UDP ports just because you have more traffic,” Register says. “How smart are you at cycling through your NAT tables? Some of the challenges that might not show up at the lower speeds show up at higher speeds. So a lot of the testing that we see is around that kind of high level stuff.”
And that’s pre-deployment testing. Post-deployment presents its own set of challenges, especially in enterprises with strict auditing and compliance requirements.
“Increasing security requirements for compliance, recording e-mail correspondence… it’s easy at 1G; at 10G or 40G or 100G it’s really, really difficult,” Registers says. “So if you want to do that you have to be able to do that kind of intelligent load balancing and filtering out of any unnecessary data so that your tools can keep up with that bandwidth.”
That challenge is exacerbated by existing filtering and analysis tools that only run at sub-10G speeds, Register says. Tapping a 40G link would require 30 or so such tools each monitoring a slice of that 40G traffic, he says.
Ixia offers a switch that sits between the production network and the analysis tools that does just that.
“They would put our switch between the production network and their tools to do the filtering – ex., seeing only the webserver traffic – to only see a very small subset,” Register says. “I can take the input stream and balance it across 32 analysis tools. I can round robin or spread that traffic across a bunch of connected tools so you can preserve your investment with existing tools. There aren’t many analysis tools that’ll run at 40G and certainly there’s nothing that runs at 100G. So a challenge we’ll have is maintaining visibility into that traffic when they do their infrastructure upgrade.”
And then, of course, there’s always multivendor interoperability challenges at 40G. In addition to clock synchronization between systems, adding features or applications could test the stability of that interoperability, Walsh says.
“Where we’re going to have problems is when people try to implement different features,” he says. “Base Ethernet connectivity will work fine – but where we’ll start to see challenges will be in RDMA, when you do lower latency stuff around RoCE (RDMA over Converged Ethernet). And when you load a special version of an SDN application that is trying to meet a specific need. When that SDN switch is plugged into the general Ethernet population, will it interoperate exactly right?”
Upgrading you network from 10G Ethernet to 40G and 100G is not as easy as swapping out switches and line cards.
40G/100G Ethernet
Several factors have to be weighed, such as synchronizing switch clocks for the higher-speeds, especially among multivendor equipment; ensuring latency remains at acceptable levels; keeping the network design and architecture optimal for 40/100G; and making sure the existing cabling infrastructure can accommodate the 4x to 10x increase in bandwidth.
One of the caveats that users should be aware of as they migrate from 10G to 40/100G Ethernet is the need to ensure precise clocking synchronization between systems – especially between equipment from different vendors. Imprecise clocking between systems at 40/100G – even at 10G – can increase latency and packet loss.
The latency issue is a bigger problem than most people anticipate, industry experts say. At 10G, especially at high densities, just the smallest difference in the clocks between ports can cause high latency and packet loss. At 40G, it's an order of magnitude more important than it is for 10G.
This is a critical requirement in data centers today because a lot of the newer innovations are meant to address lower latencies.
“Where you’re going to have the biggest challenges will be different latency configurations if RDMA (remote direct memory access) is used,” says Shaun Walsh, Emulex senior vice president of marketing and corporate development. RDMA is a low-latency, high throughput data transfer capability where application memory is exchanged directly to and from network adapters without copying it to operating system buffers.
“You see a lot more in-rack virtual switching, VM-based switching that is very application specific,” Walsh says. “New line cards in new backplane architectures mean different levels of oversubscription. There’ll be generational tweaks, configuration ‘worrying’ that has to occur. The biggest thing (testers) are running into is making sure you get the 40G you are paying for (with regard to) latency issues, hops, and congestion visibility.”
Emulex late last year acquired Endace, a developer of network performance management tools. Demand for the Endace product and the 40G capabilities of Emulex’s XE201 I/O controller are picking up as more data centers and service providers upgrade from 10G to 40G.
Walsh expects 40G Ethernet to be a $700 million market in four to five years, roughly half the time it took 10G Ethernet to reach that mark. Driving it are next-gen blade server mid-plane interfaces and architectures, big data, analytics, video and data over mobile, BYOD and high frequency trading, Walsh says.
Another challenge is readying the cabling infrastructure for 40/100G, experts say. Ensuring the appropriate grade and length of fiber is essential to smooth, seamless operation.
This is a big consideration for users because it could mean re-wiring a significant portion of their physical plant, if not all of it. That could be an expensive and disruptive undertaking.
At the physical layer, 40G Ethernet is essentially 4x10G “lanes.” But 100G Ethernet is 4x25G lanes, which will be disruptive to the 10G and 40G infrastructure.
“100G is going to be even more of a challenge because now you’re dealing with a whole new layer of physical infrastructure,” Walsh says. “You will have a whole new generation of optics, cables, everything will be a whole new generation at that point.”
Moving bits four to 10 times faster error free is a challenge in and of itself. Making sure the higher level systems – routers and switches – deliver services and non-disruptive service quality at those speeds is equally as challenging, if not more so.
Each device has to do this at one-fourth or one-tenth the time it does at 10G. For a router, it means performing all of the packet inspection, queuing, lookups, filtering, policing, prioritization, table updating and logging while meeting SLAs by not dropping or reordering packets, or increasing latency or jitter.
“Routers aren’t just forwarding packets,” says Scott Register, senior director of product management for Ixia, a maker of test, measurement and visibility tools. “There’s carrier grade NAT, application ID, and security processing and things like that. One of the more interesting testing is, what services can you enable at that rate before you start having problems?”
With carrier grade NAT, the problems get harder as that traffic load increases, Register says. In addition to throughput, increased session capacity and more concurrent connections are also issues as bandwidth climbs from 10G to 40/100G.
“You don’t get more TCP or UDP ports just because you have more traffic,” Register says. “How smart are you at cycling through your NAT tables? Some of the challenges that might not show up at the lower speeds show up at higher speeds. So a lot of the testing that we see is around that kind of high level stuff.”
And that’s pre-deployment testing. Post-deployment presents its own set of challenges, especially in enterprises with strict auditing and compliance requirements.
“Increasing security requirements for compliance, recording e-mail correspondence… it’s easy at 1G; at 10G or 40G or 100G it’s really, really difficult,” Registers says. “So if you want to do that you have to be able to do that kind of intelligent load balancing and filtering out of any unnecessary data so that your tools can keep up with that bandwidth.”
That challenge is exacerbated by existing filtering and analysis tools that only run at sub-10G speeds, Register says. Tapping a 40G link would require 30 or so such tools each monitoring a slice of that 40G traffic, he says.
Ixia offers a switch that sits between the production network and the analysis tools that does just that.
“They would put our switch between the production network and their tools to do the filtering – ex., seeing only the webserver traffic – to only see a very small subset,” Register says. “I can take the input stream and balance it across 32 analysis tools. I can round robin or spread that traffic across a bunch of connected tools so you can preserve your investment with existing tools. There aren’t many analysis tools that’ll run at 40G and certainly there’s nothing that runs at 100G. So a challenge we’ll have is maintaining visibility into that traffic when they do their infrastructure upgrade.”
And then, of course, there’s always multivendor interoperability challenges at 40G. In addition to clock synchronization between systems, adding features or applications could test the stability of that interoperability, Walsh says.
“Where we’re going to have problems is when people try to implement different features,” he says. “Base Ethernet connectivity will work fine – but where we’ll start to see challenges will be in RDMA, when you do lower latency stuff around RoCE (RDMA over Converged Ethernet). And when you load a special version of an SDN application that is trying to meet a specific need. When that SDN switch is plugged into the general Ethernet population, will it interoperate exactly right?”
