Monitor Submarine Cables

Monitor Submarine Cables

We spend a lot of time monitoring our internal networks. Obviously, this is where we have the most tools at our disposal and where our actual responsibility lies. But, to provide good service to our customers and/or end users we also need to be aware of what is happening at our Internet providers and above. If you have global services then I recommend you monitor the submarine cables as well. For example, this was the latest submarine cable damage that impacted regions in Africa. https://subtelforum.com/angola-wacs-submarine-cable-damage/ Here is a map of the cable(s) that were impacted. If you want to go further and receive notifications on your mobile device then you can use the IFTTT app I created....
Read More
Enable DNSSec

Enable DNSSec

Performance and security is always a balancing act, but in the case of DNSSec it's a no-brainer. In short, DNSSec allows a client to trust the domain owner when performing DNS queries. It’s another step to defending your domain (and subsequently your content and network) from the bad guys. An added benefit is there is no noticeable impact to performance! CloudFlare just released a great blog post on their DNSSec offerings and how they are expanding. In that post they discuss DNSSec in much more detail along with their implementation of it. After reading the post I enabled it on my site and it was very straightforward. Enable DNSSec in CloudFlare Log into your CloudFlare account Select the DNS option at the top for the domain Scroll down to the DNSSec option and click the button to enable it. Make note of the values presented for the DS Record Add the DS Record to Your Registrar In this case my registrar is NameSilo, so...
Read More

Shuffle Sharding

So, all credit goes to Colm MacCárthaigh for this one. I think his recent post on Shuffle Sharding is so go it deserves a share and a place on my blog to serve as a reminder for me from time-to-time. This is one way AWS achieves the level of reliability and stability it has for its customers. Some of the methodology can easily be applied to traditional and on-prem infrastructure though as well. Check it out here!...
Read More
Case Study: Out of Memory

Case Study: Out of Memory

Symptoms Website randomly goes down a few times a week Server stopped responding Network and CPU logs show a small spike, but not enough to lock up a server Stopping and starting the server resolves the problem Details This pattern repeated several weeks until the customer grew tired of rebooting the server. The evidence did not seem to lead to a system issue or network or security security problem such as a denial of service. The application logs were clean as well. Also of importance is that this server was a Linux EC2 instance in AWS. Troubleshooting Being that rebooting the server resolved the problem every time, it was decided to duplicate the EC2 instance from its snapshot image. This was completed quickly, but the issue appeared again that night and several times the following day. Finally, an error was seen in the system logs which pointed directly to a memory issue. Solution After this, it was discovered that the server was a T2.micro instance with...
Read More

Wireshark Webinars

I have attended multiple Wireshark webinars presented by Riverbed and leaders in the field. They title this series "Return to the Packet Trenches" with some sort of variation or subtitle for the different sessions. I always walk away with something new. This latest webinar was no exception. It reviewed several CLI options for creating, analyzing, and editing packet captures. I highly recommend attending these webinars if you have any interest in Wireshark and staring at packets. For more resources I recommend or to see the tools I've created, please look at my "Network Performance" drop-down menu at the top of this page. Here are links to their resources as sent to me in their follow-up email: Wireshark CLI tools & scripting (by Sake Blok) https://sharkfestus.wireshark.org/assets/presentations18/33.zip Presentation Video https://youtu.be/IZ439VNvJqo (1:11:14) TShark Command Line using PowerShell (by Graham Bloice) https://sharkfesteurope.wireshark.org/assets/presentations17eu/33.7z Custom LUA dissectors to the rescue in root cause analysis (by Sake Blok) https://sharkfesteurope.wireshark.org/assets/presentations17eu/21.pdf Review the SharkFest’18 EUROPE agenda and other information, For more "Packet Trenches"  resources, check out these links. Watch the replay...
Read More
Book Recomendation: “The Phoenix Project”

Book Recomendation: “The Phoenix Project”

Other than the main character being a manager, it is amazing how close this book mirrors my career path so far. This is fiction, but does a good job introducing business and cloud concepts. I would definitely recommend this for anyone in IT. The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win by Gene Kim My rating: 3 of 5 stars A story that anyone from an IT operations background can relate to. The various character personalities keep it interesting and even relatable still! It helps provide motivation to use ITIL methodologies, Kanban process, cloud computing, and more. My only critique is that it's a slow start with an abrupt end. I'm interested to read "The DevOps Handbook" now to see the real world advice and stories. View all my reviews...
Read More
Log AWS VPC Flows

Log AWS VPC Flows

As I transition to working in “the cloud” more I am embracing the new technologies and methodologies. However, I’m also trying to replicate what I do in on-prem environments when it makes sense. One way I like to collect and analyze data is using NetFlow. NetFlow provides network conversation details at a higher and summarized level. This has led to quicker recovery time on numerous occasions, or avoided issues entirely. It isn’t exactly the same, but I have figured out how to log AWS VPC flows to provide the data. Here’s a brief walk through of the setup.   Create Flow Logs The first step is to select the VPC and then the “Create Flow Log” menu item from the “Actions” drop down.   See them attached to VPC After confirming its creation I saw the log ID listed in the “Flow Logs” tab.   View Configured Flow Logs Hopping over to the CloudWatch logs I could see my newly created log group.   Log View After selecting the log, I could...
Read More
Diagram Your Service

Diagram Your Service

 I love packets and tracing issues at a micro level. However, like I stated in Preparing for the Capture you need to know where to capture before you can dig into the bits an bytes. In order to know where to capture you must understand your service/app/network. The best way to do that is to diagram your service. The Diagram The featured image on the post and the same one included below is a high level example of an architecture diagram of this blog. I use CloudFlare and AWS services currently to host it. The diagram shows this flow along with the purpose of these services and a little more detail outlining the layout of AWS. In a more detailed and private diagram I could also include breakouts showing the actual services running such as Wordpress, Apache, and MariaDB. I could also include external services that provide MFA, email, monitoring, and notifications.   Created with draw.io The Purpose An architecture diagram does more than highlight good capture points....
Read More

Command For Dozing

I generally avoid creating posts that are specific to my employer, but this is already public knowledge and it was fun to be involved even in a small way. So often us "packet junkies" only get to see the results of our work through the lens of smoothly flowing packets. If we're lucky we might hear the delight in our customer's voice over the phone or get a nice email sharing the results. Once in awhile though, we get to be a part of something a little bigger with real-world application. This was one such case. A technology called "Command for Dozing" was on display in Malaga, Spain. In simpler terms, customers were able to play with a real-life Cat bulldozer over 5000 miles away in Arizona. I will not go into specifics, but this is a perfect place to ask fellow sharks how you would help ensure smooth delivery of video and control packets across a worldwide network. You can...
Read More