*Disclaimer: all captures in this post were anonymized using TraceWrangler.
I was recently asked to help with a performance issue. I was informed a transfer was going to take weeks instead of a couple days as expected. The transfer rate was getting 80Mbps throughput max on a 10Gbps connection. So, I setup captures at both ends and got to work. This is just a quick summary of that work with the classic tell-tale signs of a performance problem.
The first thing I noticed were 30 zero window segments in a matter of seconds in the “Expert Information” window. One or two might be tolerable under normal circumstances, but 30 is something of interest. Small TCP window sizes and zero window packets generally mean there is a problem with one of the end devices.
This grabbed my attention, so I moved back to the packet list. When looking at the Zero Window packets I noticed delays anywhere from 100 to 300 ms before the Window Update. (If you don’t have a TCP Delta/Delay column setup already in your instance, I highly recommend it!)
TCP Zero Window follow by a delay
This was also clearly evident in the Time Sequence Delay graph. This is a classic example of the “stair step” graph. This should look more like a diagonal line up to the right. The receiving end cannot keep up with the data flow and is slowing the traffic.
Reviewing the Window Size graph revealed an even more disturbing picture. It seems the server couldn’t keep up at all with the incoming data. The window sizes dropped rapidly, and they all delayed before the acknowledgement and window update.
I decided to glance at the TCP options in the packets of the TCP handshake. The calculated window sizes and maximum segment size looked good. The Window scaling leaved something to be desired. Packet TCP Options
These were all classic symptoms of a performance issue on the receiving server. In this case, the server admin performed a few network tweaks and adjusted settings in the security software resulting in the disappearance of the reducing window sizes/zero window packets (the visible network symptom of root cause). Consequently, the delays were shortened and performance increased. Unfortunately, the day took a quick turn and I was unable to capture the new data to get a “before and after” snapshot or ask what “tweaks” he made specifically, but the results speak for themselves. Another performance issue resolved, another happy customer, and a cookie cutter example of performance indicators in Wireshark.
For 11 years networking was my profession with a specialized focus on proactive and reactive performance analysis. More recently I have embraced the AWS platform. This blog reflects my experience both past and present.
Like many of you, I am watching the AWS re:Invent keynotes and announcements remotely. Being network and performance focused here are the announcements I am currently most excited about. I will update this list as the week progresses.
VPC ingress routing now makes routing to virtual appliances easier! This is a much welcomed improvement as routing inside the VPC has always been a bit tricky and cumbersome. https://aws.amazon.com/about-aws/whats-new/2019/12/amazon-vpc-ingress-routing-insert-virtual-appliances-forwarding-path-vpc-traffic/
Transit Gateway Manager manages and monitors the on-prem to cloud networks centrally and easier.
Trump is the first U.S. President to bring religious freedom as a main point of discussion before the U.N. Whether you subscribe to an organized religion or not religious freedom is central to a free society. The ability to adhere to personal sincerely held beliefs is the ultimate freedom of choice. You cannot have free speech or access to any other God-given rights without first having freedom of belief, thought, and emotion.
My PC was dual-purpose for most of its life existing as a desktop PC and a server for the various functions we needed. As the PC aged I started moving apps to the cloud or other devices. I previously posted about how to Host Plex on AWS and Wasabi. That post explains some of the “whys” behind my decisions for that solution. Another central app is Ubiquiti’s Unifi Video software which runs my security cameras.
As promised, CloudFlare delivered HTTP/3 this week. If you don’t know what this is and what it could mean for web browsing in the future I highly recommend reading their blog and other sites about it. HTTP/3 brings another fundamental shift to transport protocols by building on the lessons learned from TCP and UDP. I’m particularly intrigued to see HTTP requests begin before the handshake is complete. CloudFlare released it to their customers who signed up to enable it early.
Purpose This can be used to fade sounds in/out by adjusting their volumes using a pedal or the wheel. You can even fade one sound in while fading the other out. It can also be used to apply effects, such as ‘wah’ to the pads.
The Morph allows you to control several parameters at once with one “source” You can use a foot pedal, wheel, or “aftertouch” to control different parameters or effects.
This week I attended my first AWS Immersion Day. The event was hosted by Justin McWhirter [justindm.me]. The focus of the day was serverless, and was centered around the Wild Rydes Workshops. By the end of the day we ended up with a web application that looked like this that was built upon many integrated AWS services such as Cognito, Kinesis, S3, Lambda, Amplify, API Gateway, and more. As someone who spends my time generally focused on networking within AWS this was a welcomed change of pace and a good learning experience.
This summer I have been working on recording keyboard parts for my church’s next set of worship videos. I have shared the previous worship videos, Christmas programs, a couple of covers, and originals on this site, but I have never shared any of the “behind the scenes” work. I thought I would take the time to share a sample of that now.
In this video I have panned all of the parts I created to the left ear.
There are times when you may have a need to test server performance when investigating an issue or doing a predictive analysis. I learned an easy way to do this on Linux using the built-in ‘yes’ command.
From the man page for the ‘yes’ command:
NAME yes - output a string repeatedly until killed This does exactly as it says and will consume the CPU unless it is controlled or killed.
Wasabi Storage Storage is one of the costliest options in the cloud and probably the biggest deterrent to migration. Fortunately, a handful of contenders are changing the game and breaking into affordable options for personal budgets. One of these companies is Wasabi. I have embraced the AWS platform, so on the surface this appears to be in opposition to that. Maybe it is, but Wasabi utilizes AWS S3 on the backend with a pricing strategy fit for personal as well as business use.
There are 2 imprints on the pillow my wife uses for the twins. These imprints are from their heads and bodies as they’ve grown and been laid down in the same place for months.
We are all like the pillow receiving imprints on our lives from those around us. Some people are around just a short while and we barely notice. Others we invite back repeatedly and the relationship can last for years creating deep and long lasting effects.
