I am helping out my old team, researching bugs reported by the customer. Recently one was opened that was going to be a big deal unless it got fixed pronto. The customer put this fact in the bug description. I worked with my team lead on this problem since he had some access issues on his computer.
First thing I did was get an agreement on how the application was supposed to act in the normal flow. Then I reviewed some abnormal flows. Finally we looked at the production data for the examples provided by the customer. I have been doing this for many years. So I knew how to mine our audit tables for the history of how objects flowed through the system.
My team lead was amazed by this. He apparently had never queried the database to get this information. We got a bunch of triggers which store off state changes of interest. This comes in really handy when you are debugging tough problems. I spotted the pattern. The software was actually behaving as designed. These were just some boundary cases which do not behave like the normal flow. As soon as I briefed the main customers on our findings, the trouble ticket was closed with a reason of no trouble found.
Rolling my own ICMP client - I searched around on the Internet for a program named Loki. It was supposed to send traffic using ICMP. The idea is to hide stuff in there that firewalls ...