Diagram Your Service

Diagram Your Service

 I love packets and tracing issues at a micro level. However, like I stated in Preparing for the Capture you need to know where to capture before you can dig into the bits an bytes. In order to know where to capture you must understand your service/app/network. The best way to do that is to diagram your service. The Diagram The featured image on the post and the same one included below is a high level example of an architecture diagram of this blog. I use CloudFlare and AWS services currently to host it. The diagram shows this flow along with the purpose of these services and a little more detail outlining the layout of AWS. In a more detailed and private diagram I could also include breakouts showing the actual services running such as Wordpress, Apache, and MariaDB. I could also include external services that provide MFA, email, monitoring, and notifications.   Created with draw.io The Purpose An architecture diagram does more than highlight good capture points....
Read More

Command For Dozing

I generally avoid creating posts that are specific to my employer, but this is already public knowledge and it was fun to be involved even in a small way. So often us "packet junkies" only get to see the results of our work through the lens of smoothly flowing packets. If we're lucky we might hear the delight in our customer's voice over the phone or get a nice email sharing the results. Once in awhile though, we get to be a part of something a little bigger with real-world application. This was one such case. A technology called "Command for Dozing" was on display in Malaga, Spain. In simpler terms, customers were able to play with a real-life Cat bulldozer over 5000 miles away in Arizona. I will not go into specifics, but this is a perfect place to ask fellow sharks how you would help ensure smooth delivery of video and control packets across a worldwide network. You can...
Read More
Case of the Tired Firewall

Case of the Tired Firewall

Have you ever had a nightmare where you are being chased and you can't just seem to run away fast enough? No? Well, maybe you've tried running through snow up to your knees or swimming while wearing jeans. All of those examples point to situations that feel like something isn't quite right. Cases where there could be better performance if only something was changed or improved. Sometimes this same thing happens to network devices. In this example, it's the case of the tired firewall. Problem It was a typical day in the office with users milling about drinking coffee, others happily working, and some furiously pecking at their keyboards with deadlines looming. For a small segment of users, though, things weren't normal. They were staring at their monitors with looks of puzzlement and confusion. Instead of working on their latest application update they were running tests with varying and unusual results. They found several symptoms, but were unable to pinpoint the root...
Read More
TCP Performance Options

TCP Performance Options

In order to understand application performance across the network, we first have to understand the basic mechanisms. In this case that foundation is built on TCP, and, more specifically, the built-in TCP Performance Options. There are many things that can be done in an application to improve performance. There are also several options from a network perspective, and more still in the operating systems. However, these all rely on the underlying protocol. The Warm Up This blog can serve as a warm up to understanding TCP performance. But, it is still a high-level overview and based on my knowledge and experience. So, for other perspectives I have included some other blog links below in the "Other References" section. These are blogs from some experts in the field and I highly recommend reviewing their content and subscribing to them as well. Nothing beats going straight to the source; however, so I've also included some of the RFCs pertaining to TCP performance in the...
Read More
Classic Performance Example

Classic Performance Example

*Disclaimer: all captures in this post were anonymized using TraceWrangler. I was recently asked to help with a performance issue. I was informed a transfer was going to take weeks instead of a couple days as expected. The transfer rate was getting 80Mbps throughput max on a 10Gbps connection. So, I setup captures at both ends and got to work. This is just a quick summary of that work with the classic tell-tale signs of a performance problem. The first thing I noticed were 30 zero window segments in a matter of seconds in the "Expert Information" window. One or two might be tolerable under normal circumstances, but 30 is something of interest. Small TCP window sizes and zero window packets generally mean there is a problem with one of the end devices. This grabbed my attention, so I moved back to the packet list. When looking at the Zero Window packets I noticed delays anywhere from 100 to 300 ms before the...
Read More