Case of the Tired Firewall

Case of the Tired Firewall

Have you ever had a nightmare where you are being chased and you can't just seem to run away fast enough? No? Well, maybe you've tried running through snow up to your knees or swimming while wearing jeans. All of those examples point to situations that feel like something isn't quite right. Cases where there could be better performance if only something was changed or improved. Sometimes this same thing happens to network devices. In this example, it's the case of the tired firewall. Problem It was a typical day in the office with users milling about drinking coffee, others happily working, and some furiously pecking at their keyboards with deadlines looming. For a small segment of users, though, things weren't normal. They were staring at their monitors with looks of puzzlement and confusion. Instead of working on their latest application update they were running tests with varying and unusual results. They found several symptoms, but were unable to pinpoint the root...
Read More
TCP Performance Options

TCP Performance Options

In order to understand application performance across the network, we first have to understand the basic mechanisms. In this case that foundation is built on TCP, and, more specifically, the built-in TCP Performance Options. There are many things that can be done in an application to improve performance. There are also several options from a network perspective, and more still in the operating systems. However, these all rely on the underlying protocol. The Warm Up This blog can serve as a warm up to understanding TCP performance. But, it is still a high-level overview and based on my knowledge and experience. So, for other perspectives I have included some other blog links below in the "Other References" section. These are blogs from some experts in the field and I highly recommend reviewing their content and subscribing to them as well. Nothing beats going straight to the source; however, so I've also included some of the RFCs pertaining to TCP performance in the...
Read More
Classic Performance Example

Classic Performance Example

*Disclaimer: all captures in this post were anonymized using TraceWrangler.   I was recently asked to help with a performance issue. I was informed a transfer was going to take weeks instead of a couple days as expected. The transfer rate was getting 80Mbps throughput max on a 10Gbps connection. So, I setup captures at both ends and got to work. This is just a quick summary of that work with the classic tell-tale signs of a performance problem. The first thing I noticed were 30 zero window segments in a matter of seconds in the "Expert Information" window. One or two might be tolerable under normal circumstances, but 30 is something of interest. Small TCP window sizes and zero window packets generally mean there is a problem with one of the end devices.   This grabbed my attention, so I moved back to the packet list. When looking at the Zero Window packets I noticed delays anywhere from 100 to 300 ms before the...
Read More