Problems After Updates?

One of the most common problems any IT admin faces is a software update. While software updates are generally considered a good thing, because they patch security flaws, fix bugs, try to improve performance, and more, they are also a common source of problems. Every admin knows to be ready for calls after a scheduled maintenance window. This issue was no different.   A ticket came in stating users could not access a web app through the backend system after an upgrade to Java 1.7. The server, java, and app logs all looked ok and appeared to be running properly. Also, interestingly, the web app worked when accessed directly from a web browser. This sounded like a perfect opportunity for a quick packet capture and analysis. Here is what was produced:     *Note: In order to maintain the SSL session info I could not anonymize this, so I'm just using a screen shot instead of sharing the capture on CloudShark.   I've done quite a few...
Read More

Using Fiddler to Fix Issues

I've helped many users who say Fiddler has "fixed" their issue. Unfortunately, this is a bit deceptive. Fiddler is an excellent debugging tool for web apps, but it does not permanently resolve problems. What it does do is act as a proxy with its own connection settings. This allows it to act as a "man in the middle" and even decrypt the traffic to provide better more insight into application behavior. Sometimes, this is just enough to correct the underlying problem and give the illusion that all is well. This can be very frustrating when trying to find and debug the problem! I have personally seen Fiddler "help" with the following: Proxy issues TLS versioning SSL cert problems Telerik themselves have a great post on this here outlining the technical details and corrective actions. If you do any sort of debugging with Fiddler it's worth a read. Side Note: If you help end users, but...
Read More

Packet Threat Analysis

Everyone needs to do some housekeeping at different points, and I figured it was time I did some a basic security sweep of my setup. To get started, I performed a quick packet capture on the very server that hosts this blog. I decided to give one of CloudShark's newer and more distinct features a spin with my recently created account; their Threat Assessment tool. I thought it would be interesting to pit this against PacketTotal as well. These are both great tools with similar, but also different purposes. At the time, I had SSH and web ports open along with a few other unused ports for various common services. The only true security measure in place was a few basic iptables rules. CloudShark What I Liked: Up front, quick severity level rating dashboard Brief descriptions of issues which helps puts everything in laymen's terms World map view Privacy settings External references to source data and additional information Ability to view the...
Read More
Case of the Named Pipes

Case of the Named Pipes

Problem I have come to expect vague error messages that seemingly blame the network. This one is no different. Server Error in '/' Application. The network path was not found Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.ComponentModel.Win32Exception: The network path was not found Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. (The stack trace is long and doesn't actually provide root cause either, so I omitted it to keep this post brief.) In this case, a web server was providing the error above after receiving a user request and attempting to connect to a backend database. Unless there are other dependencies, the problem is between the web server and database. Both ping tests and telnet tests to...
Read More
Case of the Tired Firewall

Case of the Tired Firewall

Have you ever had a nightmare where you are being chased and you can't just seem to run away fast enough? No? Well, maybe you've tried running through snow up to your knees or swimming while wearing jeans. All of those examples point to situations that feel like something isn't quite right. Cases where there could be better performance if only something was changed or improved. Sometimes this same thing happens to network devices. In this example, it's the case of the tired firewall. Problem It was a typical day in the office with users milling about drinking coffee, others happily working, and some furiously pecking at their keyboards with deadlines looming. For a small segment of users, though, things weren't normal. They were staring at their monitors with looks of puzzlement and confusion. Instead of working on their latest application update they were running tests with varying and unusual results. They found several symptoms, but were unable to pinpoint the root...
Read More

Case of the Rogue Server

Several hundred users lost network connectivity. They went down randomly, one by one, and over a short period of time. Some users had intermittent connectivity. All of the network devices were online and functional.  Users were roaming the halls and getting bored. This called for a packet capture, but with clients offline it had to be done on a network switch. In this instance, the capture was performed at the distribution switch on the layer 3 VLAN. It revealed clients frantically trying to connect but being rejected, dropped, and ignored. With everything down, something had to be done. The L3 VLAN was rebuilt and port security was removed. Nothing worked. An analyst at one point decided to clear the arp table. It helped momentarily, and then things fell back into disarray. That was the first real clue, though.   Viewing the ARP table showed the same MAC for several IPs including the client switch IPs. This MAC belonged to a device not managed by...
Read More

Case Study: Capture at Both Ends

While the general rule of thumb is to capture at the client, or at least start there, sometimes it's necessary to take captures at both ends of a connection. The client perspective will allow you to view the problem as it is seen from the client. The server perspective might show the same thing. Or, in some cases like this one, it will provide the reason for the problem. The problem was that a webpage wouldn't load. There were various errors and no real indication of the problem. While SSL was suspected, there was no proof. So, we started with a capture at the client. From the client, the capture reveals it sent a TLSv1.2 Client Hello as expected. However, it then abruptly ends with a FIN with no server hello. Versions and ciphers were compared in the settings, but everything matched. More data was required. Another capture point was setup on the server. Immediately, the problem was revealed. The server received a...
Read More
Performing a Capture

Performing a Capture

Start the Capture Now that you've decided where to capture, and you've prepared your interfaces and filters, you are ready to perform the capture. All you have to do at this point is hit the start button or double-click the interface in the list. There are usually multiple interfaces listed, so make sure you know which one you are wanting to use. Generally, this may be indicated by a small moving graph to the right of the device name indicating there is traffic present (the screen shot below currently show no traffic as I was in a lull). With Wireshark and some other utilities you also have the option to capture from multiple interfaces at once. Once the process is started, Wireshark uses the WinPcap driver to collect the packets from your network card and then displays them on-screen (if you haven't disabled that option for performance reasons). You will see columns and rows of numbers, letters, and words rapidly scrolling across...
Read More
Preparing for the Capture

Preparing for the Capture

Packet captures give us a very detailed and in-depth look at network traffic. They can be used to establish baselines, discover network devices, diagnose application and performance issues, or identify security threats. The previous post described what packets are and their function at a high level. It also gave an overview of the process used to capture them. Once you have identified your purpose for performing a capture, you can begin preparing for it. Caution: This is where things get technical pretty fast. A good knowledge of basic networking concepts will help from this point forward. This post will also be a little lengthier. Preparing for a capture follows the same basic process regardless of the purpose, albeit a few tweaks here and there: Select the capture point(s) Choose the capture method Apply the appropriate filter(s) and configurationPost-prep actions Begin the capture Perform the task to generate target traffic if...
Read More
Capturing Light

Capturing Light

How do you go about catching the one of the fastest things known to man (light) at a specific point in time with pinpoint accuracy over and over again? With a little patience and your network card, of course! This post is an introduction to the process of capturing network traffic (aka "sniffing" or "tracing"). With most of my blog being dedicated to network performance analysis, a post like this is foundational, and will help you understand the basics moving forward if you are new to "sniffing the wire". Networks range in size from small home networks with 1 PC to global corporations with many users and data centers, to the largest network (or collection of networks depending upon your definition) of them all, the Internet. While size, speed, design, and costs may vary, they all use the same signals to communicate. These signals may take the form of electricity (ethernet cables), light (fiber), and radio waves (wireless) across varying mediums. These...
Read More