Problems After Updates?

One of the most common problems any IT admin faces is a software update. While software updates are generally considered a good thing, because they patch security flaws, fix bugs, try to improve performance, and more, they are also a common source of problems. Every admin knows to be ready for calls after a scheduled maintenance window. This issue was no different.   A ticket came in stating users could not access a web app through the backend system after an upgrade to Java 1.7. The server, java, and app logs all looked ok and appeared to be running properly. Also, interestingly, the web app worked when accessed directly from a web browser. This sounded like a perfect opportunity for a quick packet capture and analysis. Here is what was produced:     *Note: In order to maintain the SSL session info I could not anonymize this, so I'm just using a screen shot instead of sharing the capture on CloudShark.   I've done quite a few...
Read More

Using Fiddler to Fix Issues

I've helped many users who say Fiddler has "fixed" their issue. Unfortunately, this is a bit deceptive. Fiddler is an excellent debugging tool for web apps, but it does not permanently resolve problems. What it does do is act as a proxy with its own connection settings. This allows it to act as a "man in the middle" and even decrypt the traffic to provide better more insight into application behavior. Sometimes, this is just enough to correct the underlying problem and give the illusion that all is well. This can be very frustrating when trying to find and debug the problem! I have personally seen Fiddler "help" with the following: Proxy issues TLS versioning SSL cert problems Telerik themselves have a great post on this here outlining the technical details and corrective actions. If you do any sort of debugging with Fiddler it's worth a read. Side Note: If you help end users, but...
Read More

Case of the Large Header

Just because you can do something doesn't always mean you should. One such example of this is using large HTTP headers. While the HTTP specification itself doesn't set boundaries, most web servers have default limits around 8 KB. Other devices in the path such as firewalls/WAFs, proxies, and load balancers also have similar limits.   Problem The application testers were receiving a reset error. Their application and web server logs did not show any problems.   Analysis The first question asked was, "If the web server isn't sending the reset error, what is?" In this case we found there were several devices in the path including a domain firewall and a load balancer. The firewall admin saw two-way traffic hitting an accept rule and passing through. That left the load balancer. The load balancer admin confirmed via a packet capture that it was, in fact, sending a reset near the end of the TCP stream. Why would the load balancer send a reset? A load balancer does exactly that....balances...
Read More

Packet Threat Analysis

Everyone needs to do some housekeeping at different points, and I figured it was time I did some a basic security sweep of my setup. To get started, I performed a quick packet capture on the very server that hosts this blog. I decided to give one of CloudShark's newer and more distinct features a spin with my recently created account; their Threat Assessment tool. I thought it would be interesting to pit this against PacketTotal as well. These are both great tools with similar, but also different purposes. At the time, I had SSH and web ports open along with a few other unused ports for various common services. The only true security measure in place was a few basic iptables rules.   CloudShark What I Liked: Up front, quick severity level rating dashboard Brief descriptions of issues which helps puts everything in laymen's terms World map view Privacy settings ...
Read More
Case of the Named Pipes

Case of the Named Pipes

Problem I have come to expect vague error messages that seemingly blame the network. This one is no different. Server Error in '/' Application. The network path was not found Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. Exception Details: System.ComponentModel.Win32Exception: The network path was not found Source Error: An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below. (The stack trace is long and doesn't actually provide root cause either, so I omitted it to keep this post brief.)   In this case, a web server was providing the error above after receiving a user request and attempting to connect to a backend database. Unless there are other dependencies, the problem is between the web server and database. Both ping tests and telnet tests to...
Read More
Case of the Tired Firewall

Case of the Tired Firewall

Have you ever had a nightmare where you are being chased and you can't just seem to run away fast enough? No? Well, maybe you've tried running through snow up to your knees or swimming while wearing jeans. All of those examples point to situations that feel like something isn't quite right. Cases where there could be better performance if only something was changed or improved. Sometimes this same thing happens to network devices. In this example, it's the case of the tired firewall. Problem It was a typical day in the office with users milling about drinking coffee, others happily working, and some furiously pecking at their keyboards with deadlines looming. For a small segment of users, though, things weren't normal. They were staring at their monitors with looks of puzzlement and confusion. Instead of working on their latest application update they were running tests with varying and unusual results. They found several symptoms, but were unable to pinpoint the root...
Read More

Case of the Rogue Server

Several hundred users lost network connectivity. They went down randomly, one by one, and over a short period of time. Some users had intermittent connectivity. All of the network devices were online and functional.  Users were roaming the halls and getting bored. This called for a packet capture, but with clients offline it had to be done on a network switch. In this instance, the capture was performed at the distribution switch on the layer 3 VLAN. It revealed clients frantically trying to connect but being rejected, dropped, and ignored. With everything down, something had to be done. The L3 VLAN was rebuilt and port security was removed. Nothing worked. An analyst at one point decided to clear the arp table. It helped momentarily, and then things fell back into disarray. That was the first real clue, though.   Viewing the ARP table showed the same MAC for several IPs including the client switch IPs. This MAC belonged to a device not managed by...
Read More
Become a Certified Wire Shark (WCNA)

Become a Certified Wire Shark (WCNA)

I recently sat for the Wireshark Certified Network Analyst (WCNA) certification again. This will be the second time I have taken it and the second time I have passed. I have taken several various networking certification exams, networked with people who have sat for others, and read about many more. Keeping all of that in mind, I think this is one of the most straightforward certification tests I have seen. Laura Chappell, Gerald Combs, and the team have done a great job with the books and preparation materials. If you're wondering if the exam is right for you, please continue.   Who should test? Network Admins If you are a network admin or analyst in any capacity then this certification will add to your resume and give you an edge over your peers. More than that, it will teach you about the protocols and applications that traverse your network and their expected behaviors. How can you architect and maintain a road if you don't...
Read More

Rename Files to WS File Set Format

Using file sets in Wireshark is a great feature. It allows for quickly navigating between smaller files instead of experiencing sluggish performance when analyzing one large file. However, there are times when packet captures were taken using a system other than Wireshark (such as TCPDump or Dumpcap). Other times someone else performs the captures and uses a different naming convention. Either way, there are times when it would be nice to convert these names into Wireshark's file set naming convention. For a full write-up on the function and naming convention, please see Wireshark's documentation here. To get started renaming files, please see below. Using Windows PowerShell: Create a folder where you want to rename files Create a new powershell script file with .ps1 extension (i.e. rename.ps1) Use the following script: 4. Run the script by executing it in PowerShell or right-clicking on it and selecting "Run with Powershell"  Disclaimer: I'm relatively new to PowerShell, so...
Read More

Case Study: Capture at Both Ends

While the general rule of thumb is to capture at the client, or at least start there, sometimes it's necessary to take captures at both ends of a connection. The client perspective will allow you to view the problem as it is seen from the client. The server perspective might show the same thing. Or, in some cases like this one, it will provide the reason for the problem. The problem was that a webpage wouldn't load. There were various errors and no real indication of the problem. While SSL was suspected, there was no proof. So, we started with a capture at the client. From the client, the capture reveals it sent a TLSv1.2 Client Hello as expected. However, it then abruptly ends with a FIN with no server hello. Versions and ciphers were compared in the settings, but everything matched. More data was required. Another capture point was setup on the server. Immediately, the problem was revealed. The server received a...
Read More
12