Are there any good books on debugging? I think debugging is one of the key skills of a good developer, but I have never seen a book (or blog) especially about debugging code and distributed systems.
While there are great options mentioned here, I would actually be suggesting that you stick to a very good systems book. A standing knowledge of lower level systems and instruction code will give great depth to your debugging skills. Nothing teaches reverse engineering better than experience.
At CMU, we used CS:APP (Computer Systems: A Programmer's Perspective). It's written by Randy Bryant, the same professor who used to teach it. I still have that book, it's updated for Ivy Bridge, and I truly believe that this book changed my approach towards debugging more than any other text. It will push you to the next curve.
Towards the end of the book you'd find some programming labs. I highly suggest you do them religiously. Many great men have failed at them.
I've read a number of books on debugging, it's something of an interest of mine. The best general-purpose debugging book I've read is the cumbersomely named
Debugging: The 9 Indispensible Rules for Finding Even the Most Elusive Software and Hardware Problems by David J. Agans
"Debugging Applications" by John Robbins is an old MS Press book, now 43 cents on Amazon, that covers everything from high level code structure to a nice primer on reading x86 assembly. I have more recent books, but this is the one that taught me to gird my coding loins for a fight.
Giving tips very much depends on the language you're using and the capabilities of your debugger, having worked on large legacy VC++ projects knowing the "bad smells" and common mistakes to spot is helpful, but you'll struggle to debug projects you don't have a solid knowledge of.
To improve, try learning cracking; https://microcorruption.com/ is a CTF game based around gaining access to embedded devices, it's rather fun too. I think what the most common mistake I find is two programmers having different assumptions on how some subsystem should work, so try improving your working memory as being able to remember the seemingly ok code 10 frames ago that conflicts with what's in front of you could be the key.
You need to understand how your system works and then get a good tool that allows you to observe its behavior in as much detail as possible.
I don't believe books are likely to be particularly relevant with the diversity of tools and platforms. Google the platform name plus 'debugging tool'. Then practice debugging.
If you have to, step through every single line of code and observe the program state closely at each step.
What kind of distributed system is it exactly? If it is a highly distributed system you can mock parts of it or find/ build a tool for fine-grain inspection and integrated reporting of the state of the system.
A book is not going to be able to keep up with platform and tool development.
You want to learn to Google, find good tools, practice a lot.
Somewhere on the internet the book "The Medical Detective" by Bertone Roueche should be a really good read about debugging. But I did not find a place on the internet where you can buy it. Here is a link to goodreads: http://www.goodreads.com/book/show/106164.The_Medical_Detect...
oh yes ! heartily recommended. i have a copy from amazon, and the stories are just incredible. it is just like a magicians trick. you know it's a trick because laws of physics and all that, but quite fun to deduce.
the writing style is very engaging. some stories are fully resolved, others just conclude with a plausible hypothesis that explains the data, but due to lack of sufficient evidence cannot be confirmed.
Surprised this wasn't already mentioned, but the best description I've ever read of debugging is in Zen and The Art of Motorcycle Maintenance. (Unsurprisingly, it's about debugging motorcycles, but the author was a programmer so it's directly transferable)
Robert Ward, "Debugging C". There is a second edition with a somewhat different title that I have not read, but expect would be good.
He emphasizes the use of the scientific method - observe your program's behavior, form a hypothesis as to what the bug could be, design an experiment to test your hypothesis. Just learning to be methodical rather than thrashing around mindlessly helped me tremendously.
It's out of print but I expect you could find it on Alibris.
Debugging with GDB: The GNU Source-Level Debugger by Stallman, Pesch, and Shebs http://www.amazon.com/Debugging-GDB-GNU-Source-Level-Debugge...
The Art of Debugging with GDB, DDD, and Eclipse by Matloff & Salzman http://www.amazon.com/gp/product/1593271743