*Disclaimer: all captures in this post were anonymized using TraceWrangler.
I was recently asked to help with a performance issue. I was informed a transfer was going to take weeks instead of a couple days as expected. The transfer rate was getting 80Mbps throughput max on a 10Gbps connection. So, I setup captures at both ends and got to work. This is just a quick summary of that work with the classic tell-tale signs of a performance problem.
The first thing I noticed were 30 zero window segments in a matter of seconds in the “Expert Information” window. One or two might be tolerable under normal circumstances, but 30 is something of interest. Small TCP window sizes and zero window packets generally mean there is a problem with one of the end devices.
This grabbed my attention, so I moved back to the packet list. When looking at the Zero Window packets I noticed delays anywhere from 100 to 300 ms before the Window Update. (If you don’t have a TCP Delta/Delay column setup already in your instance, I highly recommend it!)
TCP Zero Window follow by a delay
This was also clearly evident in the Time Sequence Delay graph. This is a classic example of the “stair step” graph. This should look more like a diagonal line up to the right. The receiving end cannot keep up with the data flow and is slowing the traffic.
Reviewing the Window Size graph revealed an even more disturbing picture. It seems the server couldn’t keep up at all with the incoming data. The window sizes dropped rapidly, and they all delayed before the acknowledgement and window update.
I decided to glance at the TCP options in the packets of the TCP handshake. The calculated window sizes and maximum segment size looked good. The Window scaling leaved something to be desired. Packet TCP Options
These were all classic symptoms of a performance issue on the receiving server. In this case, the server admin performed a few network tweaks and adjusted settings in the security software resulting in the disappearance of the reducing window sizes/zero window packets (the visible network symptom of root cause). Consequently, the delays were shortened and performance increased. Unfortunately, the day took a quick turn and I was unable to capture the new data to get a “before and after” snapshot or ask what “tweaks” he made specifically, but the results speak for themselves. Another performance issue resolved, another happy customer, and a cookie cutter example of performance indicators in Wireshark.
For 11 years networking was my profession with a specialized focus on proactive and reactive performance analysis. More recently I have embraced the AWS platform. This blog reflects my experience both past and present.
This summer I have been working on recording keyboard parts for my church’s next set of worship videos. I have shared the previous worship videos, Christmas programs, a couple of covers, and originals on this site, but I have never shared any of the “behind the scenes” work. I thought I would take the time to share a sample of that now.
In this video I have panned all of the parts I created to the left ear.
There are times when you may have a need to test server performance when investigating an issue or doing a predictive analysis. I learned an easy way to do this on Linux using the built-in ‘yes’ command.
From the man page for the ‘yes’ command:
NAME yes - output a string repeatedly until killed This does exactly as it says and will consume the CPU unless it is controlled or killed.
Wasabi Storage Storage is one of the costliest options in the cloud and probably the biggest deterrent to migration. Fortunately, a handful of contenders are changing the game and breaking into affordable options for personal budgets. One of these companies is Wasabi. I have embraced the AWS platform, so on the surface this appears to be in opposition to that. Maybe it is, but Wasabi utilizes AWS S3 on the backend with a pricing strategy fit for personal as well as business use.
There are 2 imprints on the pillow my wife uses for the twins. These imprints are from their heads and bodies as they’ve grown and been laid down in the same place for months.
We are all like the pillow receiving imprints on our lives from those around us. Some people are around just a short while and we barely notice. Others we invite back repeatedly and the relationship can last for years creating deep and long lasting effects.
I was feeling a little snarky today and this realization hit me:
Facebook is like a public restroom
Here me out:
Facebook is basically a public service that everyone “needs.” Some people refuse to use it. Others only use it when they are forced to, and others actively seek it out and use it routinely. Even those that don’t use it still can’t completely avoid it as everyone talks about it, they see its ads and logo everywhere, etc.
As I’ve stated in previous posts, I currently use CloudFlare as my CDN. There are several reasons for this that I won’t go into now. One of the “ToDo’s” on my list has been to clear CloudFlare’s cache when I upload new content to my blog. I was finally able to spend some time and get that done.
CloudFlare API To start things off I reviewed the doc for the CloudFlare API.
The Nord Stage 2 has four audio outputs as shown here:
You can send the various sections (synth engines) of the Stage 2 to the different outputs. This allows for more control over the mix and EQ at your output whether it be a recording bus, personal monitor, or live sound board. It’s a rather simple process.
Shift -> System Use Program Page buttons to go to Synth Outputs Scroll to change to the desired output
Having over 12 years of keyboarding experience I’ve played on numerous boards and have come to realize the various differences and nuances between the types. There are formal definitions out there and this list is probably not exhaustive, but it reflects my personal opinion. If you are reading this and are unfamiliar with keyboards this is a good place to start.
Midi controllers: Make no sounds on their own. Plug them into another device such as a keyboard or laptop to generate sounds.
It’s important to know your limits. In this case study we find a situation stemming from SMTP being throttled. This is part of the packet capture I received:
The top lines show the previous conversation ending. SMTP successfully sent 3 messages. After the 3rd message the mail server stopped responding and retransmits began. This pattern was repeatable. More than that it was repeatable from other EC2 instances. The only thing between the EC2 instances and the mail server was a router and a firewall.
If you refer to this post, you’ll see that one of my objectives for this year was to develop an Alexa app for my kids. Well, I am happy to report this objective as completed. The cover art and the image below show the high level architecture. The app idea actually started based on something I was doing for my kids that they really took a liking to. Unfortunately, for this post it might be an idea that I could actually publish and potentially monetize.
