I built it and tried it out a bit with Godot on Linux. It seems OK (the UI is a bit on the "- how many widgets do you want? - yes" side), but also a bit janky. Trying to change the font for the editor didn't work, hovering over a variable to see its value either does nothing (but there is a sub-second cursor change that indicates something is supposed to happen) or it shows an error from GDB about trying to use an expression with a type or keyword (so there was an intent to show a value on a tooltip, it is just broken) - doubleclicking on a variable does add it in some panel with its current value and a timestamp, so the functionality for reading values/expressions from the UI is there too, just not done in the same way as the tooltips.
If polished a bit it could be useful, though from all the frontends i've tried the one i disliked the least (none are great) is Gede[0] (which i just noticed had a new release a few hours ago) as it has a very simple and straightforward UI and while it doesn't expose much functionality, what exposes seem to work fine without bugs.
Thanks for your honest review. Can you explain more about not being able to set the editor font? My tests show that as working. Make sure to "Save Configuration..." to make things permanent.
Also, for the "show variable value on hover", I'll test it.
Can you (or others) create an issue on my github page for any bugs or feature requests?
> it has a very simple and straightforward UI and while it doesn't expose much functionality, what exposes seem to work fine without bugs.
Nice one, I will add it to my notes to use it next time I need debugging. The least thing I want when looking for a bug in my own code is to have to deal with bugs in the debugging tools.
Just grabbed the source, and it's a pretty bog standard ./configure ; make. Simple build instructions in the readme. Not sure this is a daunting challenge for any even moderately skilled developer.
As long as the number of programmers is growing rapidly, it’s likely that most people who use any language are probably students. The distribution automatically skews toward beginner.
That has no bearing on whether we should have good tools for skilled developers, including debuggers & makefiles & source distributions. Eventually, students become moderately skilled. And we used gdb in class when I was a student.
> That has no bearing on whether we should have good tools for skilled developers, including debuggers & makefiles & source distributions.
Unfortunately, it's also the best explanation as for why we don't have them in practice, and why all software seems perpetually developed by fresh juniors (because it is).
I don’t quite understand where this is going nor why. This little sub-thread stemmed from a good tool that does already exist, and from unsubstantiated claims that C devs are students and that the simple build steps are too hard for most people. Sure, a lot of software is written by junior devs but that has no bearing on whether gdb exists, or even how hard it is to build. We have lots of good debuggers, and luckily it only takes a few experts to write them, which is why we do have them in practice.
> This little sub-thread stemmed from a good tool that does already exist, and from unsubstantiated claims that C devs are students and that the simple build steps are too hard for most people.
It's next to impossible to properly source any claim in this area; as for a lighter standard of substantiation, C wasn't a big deal in university courses 15 years ago, so it's unlikely to suddenly become it now.
N=1, but back then, between C++ and Java courses jumping straight to IDEs (to focus on language instead of build steps), and Unix/C course sticking to direct calls to GCC, at least my year at my uni managed to go through 5 years without much exposure to make and Makefiles...
Anyway,
> Sure, a lot of software is written by junior devs but that has no bearing on whether gdb exists, or even how hard it is to build.
But it does have a bearing on GDB evolution and GDB GUIs, which almost universally expose less than the bare minimum of useful GDB features; of those that expose more, I'm yet to find one that works.
> We have lots of good debuggers, and luckily it only takes a few experts to write them, which is why we do have them in practice.
Well, yes. WinDbg, that debugger in Visual Studio, etc. :). GDB, too.
My point here is that, for better or worse, size and type of the target audience determines how much and what kind of attention the project gets. "Skilled, experienced developers" are a small niche; the vast majority of developers are fresh juniors (per the growth argument), creating pressure to satisfy them at their level. Which, for GCC, I guess it means the build steps remain arcane even relative to other GNU projects, and powerful GUI frontends for it are not a thing.
Last time I tried to build GDB from source, which was some two months ago, it wasn't in any way simple. GDB comes embedded in some GNU binutils repo, instructions to build it in isolation weren't obvious.
I ended up creating a new VM with a more recent Linux distro, that came with newer GDB, and migrating everything I'm working on to it, because that was much easier than building GDB from source.
I'm not sure where you've been looking, but source tarballs have been available from both gnu.org and the GDB website for over 20 years, with every major release going back 30 years available for download.
This has definitely been an issue for me before, I think at least for a while Ubuntu/Debian didn't ship gdb with tui enabled, you had to build it yourself.
After trying many frontends for gdb I find that the TUI is the best. You just need to know about Ctrl + L to redraw if your program is printing stuff because the interface then become garbled.
I like to do that as well. Just want to keep it short :)
Also I use vim mode for bash and since it's in .inputrc I also have a vim mode in gdb which I like a lot even though it's as good as zsh's
I've used "gdb-dashboard" a lot, would recommend it. It's similar to the TUI (though I haven't used the TUI much), but you can pick and choose to display a large variety of information and the colors make the output much easier to read.
You can also make the dashboard display on another or across multiple terminals, letting you create a much nicer window layout. I've scripted this up with tmux before to have it automatically create the terminal layout and connect them to gdb, you can create really nice layouts that way (though it can be a lot of effort).
This is usually not how wsl stuff is implemented. Usually applications set up the Windows to WSL communication themselves including Microsoft products. Might be different in this case of course but then the experience would be a lot less smooth than what Microsoft themselves set as the default with the VSCode remote extension.
I keyed off “Linux and/or wsl” specifically. I didn’t know wsl had that interoperability cooked in, I would have assumed the vm still needed gdbserver running in order to connect for a “remote” debug session.
I like how gdb with qt creator works without any complicated configuration. You click on some breakpoints then hit play and the ide takes care of the rest.
To add on to the pile of more GDB GUIs, here's one I've made: https://github.com/dzaima/grr. Though it's still got a fair bit of missing features that may be essential for some uses (due to my usage being largely assembly-level, which doesn't need too many fancy features).
Ever since the advent of LSP, Emacs has felt superior to everything else. I have no reason to leave it. Especially once they made it faster with native comp.
Like why should I keep trying this month's new editor with a couple new gimmicky features, when I can just pop a plugin onto Emacs that adds that exact feature set, while maintaining everything else how I like it.
I first really got into coding when Atom was a thing, and then that died off and became VS Code and I was pretty sad about it, because while VS Code is good, it doesn't follow the same philosophy as Atom. But then I took the time to learn Emacs ~4 years ago, and nothing new ever comes close to convincing me it's outdated tech that I need to move on from.
That was a random rant, but I'm just really appreciate Emacs, and I'm glad it's stuck around.
Yeah I first gave it a try before LSP and I don't think I was ready to be redpilled yet, because it didn't stick. So I can't comment on the state of it before then. I kinda joined once the LSP stuff got fairly smoothed out.
I prefer the GDB Graphical Interface in Emacs[1] (M-x gdb), rather than the more basic integration via GUD[2] (M-x gud-gdb). I’ve had to switch to GUD to run lldb recently, and I miss having dedicated windows that show breakpoints, threads, the current stack, etc.
The one nice thing about GUD is that the interface is consistent across debuggers, so I don’t need to refresh myself on the keyboard shortcuts when switching between debugging Python with pdb and C++ with lldb.
dape ( https://github.com/svaante/dape# ) is a great option for languages with debuggers implementing the Debug Adapter Protocol. Combined with debugpy it's killed M-x pdb for me, and has a very similar UI to M-x gdb.
DDD was taught to me when in University, 20 years ago, and it already felt clunky, my views are now much more moderate but Motif still feels like an eyesore.
Conversations over the years have shown me that DDD was a great inverse marketing tool, ironically pushing developers towards the embedded debugger UI in their favorite IDEs... despite DDD itself being indeed very powerful. But even "usefulness over aesthetics" has its limits!
There's one DDD feature that I haven't found elsewhere: its graphical representation of a struct and its contents. You can double-click on a pointer field and then it draws whatever that field pointed to, with a nice arrow connecting the two.
I've found it a very powerful yet compact way to visualize the state of a program when debugging.
There was even a story, that (at least for Common Lisp), you can start from almost blank state, but have an exception handler installed (that can continue), so as you go you live-edit and add pieces missing, or if code crashes change.
This is all good, until nowadays, where you really want to know what's deployed in production, and not just the last stuff I've live fixed.
I mean, I guess both have values tbh, but hard to pull two models like this and use... bit like - debugger or printf statements (or both!)
Current DDD under the updated OpenMotif with TTF fonts can look much better than it did in the 90's and 00's, miles ahead than LessTif/former propietary Motif. It blends perfectly with
EMWM where I have Liberation Sans/Mono for almost everything.
Since the article was written, the maintainers fixed the issues I pointed out. No need for many of those workarounds now. Versions 3.4.0 and 3.4.1 are substantial.
Is there any URL that lists what's new in v3.4.0 and v3.4.1?
The DDD website ( https://www.gnu.org/software/ddd/ ) points to the source tar.gz and the full manual, but nothing that says "What's New" in recent versions.
Might not be as direct as you'd like, but all well managed¹ GNU projects ship a NEWS file in their tarball. In this case you can also read it from savannah² without fetching the tarball.
For me it reminds me of Insight, which was my favorite GDB front-end and worked great on Windows back in the cygwin era. Unfortunately it's definitely no longer maintained. https://sourceware.org/insight/screenshots.php (Well, I guess someone did rescue it to put it on github and do some work: https://github.com/antony-jr/insight )
DDD is great. I still use it, but I am a fossil. I sought out DDD when I was looking for something similar to dbxtool, which I used on the early Sun Microsystems machines. Folks today are spoiled with things such as Source Level Debugging.
The GNU project (and RMS) seem to get a lot of meme-ridicule, but GDB is a powerhouse. I've only had occasion to mess with it a little, but it seems to have had such a huge impact on people's development work over the years.
The GNU/Linux copypasta thing which is all over reddit all the time, for example. Saying the software is "bloated" is another trope. Thirdly, a suggestion that it (GNU, free software) is all very impractical or "backward". I guess in general a feeling that it's not "cool" or "hip", maybe?
I don't know, it seems very common to me. Have you not seen anything like that?
People on Reddit have call bin utils and core utils bloated? I sort of assumed we all just took those for granted as rock solid and the reason that we can have a free Unix operating system. Huh. Maybe I'm wrong though. What subreddits are saying this? Again, genuine question, I think I must be getting old!
I'm not on Reddit these days, and haven't been in a year or so maybe, so couldn't show you anything more than what you could find with a quick search on maybe r/linux or similar heavily populated subreddits related to these things.
The point being made in the comment linked to, briefly, is how non-SW engineers "turn their nose up at open-source solutions". It's a bit broader than what I was saying, but was roughly the kind of trend I was trying to allude to.
Ok, that's interesting to hear. More reason to spend (even) less time on Reddit and Youtube, I suppose, which is probably where I've been over-exposed to these sorts of points.
When I used to program in C++ on Linux 10+ years ago, I used Qt Creator which has a built-in debugger (GDB frontend). It worked great and I don't see a reason to use anything else for C++ [and Qt].
I sometimes need to use gdb to investigate bugs in C or Ada, but it is not my main activity. As a result I will not invest days to setup a debugging environment that I will not remember how to use 6 month later. My solution: I use emacs and have a short note with instructions:
M-x gdb -i=mi exe_full_name -p 29123
M-x gdb-many-windows
set follow-fork-mode child
Has anyone found a reliable way to use a debugger when you have a) multi-process b) multi-threaded c) async d) timeouts? I would love to use a debugger but printf and logs “just work”
I used GDB to debug contributions to the ION Delay-Tolerant-Networking stack, which is composed of dozens of processes. After some practice it became super easy to attach GDB instances to several running processes and watch the C structs be passed between them.
Eclipse CDT is a whole IDE; is rather complex, and one might even say convoluted; and is Java based. Seer is none of these things.
Of course - it's not like a GDB GUI is a novelty in itself, there are quite a few. But a GDB-GUI-only utility is a meaningful and important niche to consider.
I've used classic eclipse-cdt for over a decade until it cannot keep up with post-C++14 standards. It seems that it recently regains activity with the introduction of https://github.com/ghentschke/eclipse-cdt-lsp
On a general note, I would recommend any new (and experienced!) programmers to master the debugging tools of their ecosystem. I've seen countless experienced developers use printf-based debugging and waste hourse debugging something which could've been easily figured out by setting a breakpoint and stepping through your code. This is also a good way to understand code you're unfamiliar with.
This is one area where I believe a GUI tool is so much better: I can hover over variable names to view their values, expand and collapse parts of a nested structure, edit values easily, and follow execution in the same environment I write my code in.
Sure, it doesn't help much for some scenarios (one I've heard people mention is multithreaded code, where logs are better?), but for most people it's not that far from a superpower.
My experience is the opposite: I see developers waste hours stepping through their code a line at a time when a few judiciously placed logs (printfs() are fine, but we can do better) would have told them exactly what they needed in a jiffy.
If you have a fairly shallow bug, that is a single point in your code that always behaves incorrectly, then I find debuggers reasonably effective.
But most of the bugs that I see aren't that shallow, with code misbehaving when the context is just so and perfectly fine otherwise. In those cases, I need to see lots of different invocations and their context. The debugger is like trying to drink the information ocean I need through a straw. A mostly plugged straw.
I wonder what makes our experiences so different? Do you unit test a lot? Particularly with TDD? I am guessing that this practice means I just don't get to see a lot of the bugs that a debugger would help me with.
(And it doesn't mean I never fire up the debugger. But it is fairly rare).
I have more or less the same experience like you. Logging is a very resilient and adaptable technique – I can use it on my laptop or on remote HPC clusters, almost regardless of programming language (except maybe Haskell), it works fine on parallelized code, and so on, with very little configuration needed. It’s also important to me that it can be done “async”, since some of my larger codes can only be run on HPC clusters by putting a job in a process queue and waiting.
I’ve tried debuggers and see the appeal but I find it less useful than print debugging / logging.
I also rely heavily on unit tests when writing new code, so that also reduces the surface that I need to look for bugs based on the log. Moreover, most of my projects have 1-3 programmers and can largely “fit in my head” (<10,000 lines of code), so it’s probably different if you work at a FAANG company or something.
I think you have a great point here. Debugging tools make you dependent on a particular environment. Printing based debugging can work pretty much everywhere. If you master printf programming you can solve any debugging task.
Yes, portability and simplicity are the best parts of printf.
> If you master printf
The skill ceiling is low. Printf only does so much.
You could rope in environmental optimization to the skill discussion -- the ability to isolate areas of functionality, replicate problems, reason about unknown state, and do the legwork so that you can quickly spin the increased amount of iteration required by a simpler debugging tool -- but by then you have thoroughly sacrificed both simplicity and portability and are far past the skill floor of a debugger.
If we assess this by looking for problems created by overcommitting to one approach or another, overcommitting to a debugger looks like burning time trying to get tooling to work on a problem that doesn't really need it while overcommitting to printf looks like spending way too much time iterating on tiny steps that could have been jumped over given better visibility. I've seen both, of course, but I tend to see more of the latter and more denial about the latter. When you're burning time fighting tools it's obvious. When you're burning time because you don't know how a tool could have saved you time, it's less obvious.
> programmers to master the debugging tools of their ecosystem. I've seen countless experienced developers use printf-based debugging and waste hourse debugging something which could've been easily figured out by setting a breakpoint and stepping through your code.
If you're wasting hours with printf-based debugging, I don't think you've 'mastered the debugging tools of the ecosystem'.
There are multiple ways to debug - step debugger tools, printf, logging to a file, etc. Each have their place.
If you're spending hours on any one approach, and perhaps that's the only approach you know, that's a red flag.
If you've spent hours going through printf, logging and step debugging and STILL don't have a good answer... bring in external eyes.
I've found/fixed bugs in a few minutes because of adding some log stuff first, because in those cases, it's the easiest approach. In other cases, running a debugger and setting a couple breakpoints is indeed the easier approach to start with, and I've done that.
Sometimes you find it with the first approach, sometimes you need to try the next approach.
I would guess longer compile would encourage breakpoints over printf and this would be programming language specific.
Being able to change breakpoints at runtime helps a lot when tracking down something more complex. Visual Studio breakpoints are great, and they’ve added conditional breakpoints which are even better. Previously I would approximate this by having code specifically branch to hit a breakpoint, ‘if (X) { breakHere();}’
I write a fair amount of native C++ code but only call it from either Python or dotnet so when I make a mistake here it’s usually a segfault / memory access issue which kills the process. There might be a way to debug the C++ from dotnet or Python but logging to std out helps me isolate the location of the issue which is sufficient. It’s not a big enough problem and I worry that either writing tests in C++ or learning a native debugger will pay off in time saved.
The big ones for me with log/printf debugging are:
- I can get a good idea of the temporal behavior of the program, and I can just scroll up to see earlier state, rather than having to restart the program in the debugger. (I know that "time travel" debuggers exist, but I've found them finicky.) I can scrub back and forth through time just by scrolling.
- I can compare runs by diffing the logs. Sometimes that alone is enough to show where things start going amiss. Or I can keep instrumented logs from baseline runs.
- If there's a personally useful set of printf statements in an area that I'm in a lot, I can save those off to a patch file or a local branch. I don't have to reapply my breakpoints / watchpoints in the debugger each time. Easy persistence.
(That said, I do like to start with a debugger when tackling reproducible crashes.)
A good debugger can provide more than just stepping thru code.
In IntelliJ with Java, you can set conditonal breakpoints with complex evaluations, you can set filters (only hit a breakpoint depending from where it is being called), use exception methods that only hit on certain exceptions instead of a specific line code, you can also use logging breakpoints, that act like printf debuging, but you don't need to scatter your code with print statements all over the place.
You can group, add descripitons, disable, enable and add temporary breakpoints, they are pretty powerful! I just wish intellij had a time travel debbuger like Visual Studio Pro.
Undo captures everything the process does, below the JVM level, so you can reproduce / rewind any problem you record as many times as you want (and copy the recording out of production onto a dev machine to debug, etc etc).
I agree with both of you. Printf is not enough, breakpoints are not enough. The solution lies between. Ability to gather rapidly relevant information to converge on wrong states.
ps: I wish I could work on a porcelain layer to manage the breakpoints in a more logical manner. Considering a problem you'd like to create different sets of breakpoints, run various tests and gather results to reviews. With the ability to add or remove layers rapidly. It's probably not too hard to do.
I have found combining these things to be useful: breakpoints that print stuff and auto-resume the program. Allows you to attach trace points at-will without requiring a recompile or losing state.
Yes exactly, and I'll probably say very generally that I usually use breakpoints when I am in the exploration stage of a significant state bug, and I'll usually use logging when I generally know where the bug should be but I need to pin point the exact place
I've been, and still am, at the mercy of both of printf-debugging style and real debugger.
Long time ago, worked on a port of game from PC to Playstation 1. Since we had the Yaroze "devkit" (not really a devkit, rather amateur kit for doing games), printf debugging was the only thing available.
Things kind of worked, but when we #ifdef-out the printfs it was crashing (and no debugger). We somehow discovered that one of "printf" side effect was clearing the math errors.
Interesting. I had a similar experience: once the debugging instrumentation with #ifdef macros was switched off, the code that worked before suddenly crashed. In my situation it had to do with the stack, because my debugging macros used some
local/"automatic" variables, and that had concealed the bug before in the DEBUG build.
One thing I also noticed is that using "problem-oriented" languages like Python or Java changes where you spend your time trouble-shooting: ironically, not where the problem is (business logic) anymore, those parts of the code indeed tend to work better, but intead you waste time with libraries (Java: CLASSPATH, Python: packages, all:version conflicts). In Contrast, in C/C++ it was mostly memory management errors and bugs in the actual business logic (the former is also a great distraction, somewhat diminished by the introduction of smart pointers).
At Google, had to do Java on borg, and used few times the "Cloud Trace" debugger (not sure how it was called), but it allowed you to watch multiple instances of your binary in production, and then add some isolated set of java statements around code blocks, this way you can say (somehow) - if you end up on this line, then "inject" (somehow) something to log out... and then you can add whatever you want to be logged (like arguments, variables around, etc.).
But then later it got scrapped, or something like it.
Cloud "debugging" when you have multiple instances is one of the cases where there is no suitable enough debugger (yet).
It depends on the code as much as anything. I wrote a regex engine in Zig, and the instant I get a bug report I set breakpoints on a failing test and step through.
On the other hand, I'm working on an interactive application, and when I see a problem with it, I add more logging statements until I figure out what the problem is. Any time the logs have excessive detail as a consequence, I gate them behind an 'extra' flag on a per-unit basis, only removing the ones which amount to "got here".
If I had to pick one technique, it would be logs. I naturally think in terms of a trace through the execution pathway, rather than a step-by-step examination of the state of a small window into the code. It clearly works the other way around for some people.
One thing that makes this approach better for me is that debug logging is literally free, Zig uses a lazy compilation model so logging code which doesn't apply to a release compilation isn't even analyzed, let alone compiled, let alone included. In a language which doesn't work that way, there's motive to use printf-only debugging, and clean up after yourself afterwards, and that's extra work compared to firing up a debugger. So it shifts the balance.
Another thing to consider and is important to me - logging objects and state isn't always so simple. It can often be easier for me to open the debugger to look at the state of an object which cannot easily be printed.
An object with many fields (in a language with no conveniences for it).
An object tree with multiple levels of nesting.
A list or dictionary of such objects.
In general, print-based debugging requires a greater degree of specificity. If you know exactly what you're looking for it's great.
If you are performing a more exploratory sort of debugging, a decent graphical debugger will save you a ton of time.
In my case it's basically everything since I work in java, jackson's object mapper can easily get stuck or deserialise something incorrectly if the class hasn't been annotated correctly. So it's simpler for me to pull up the debugger and I can see the "actual" data thats making up my object, it also lets me run "queries" against anything of interest (i.e. computed fields).
The default toString method I've found to be useless almost every time I wanted to inspect an object in our codebase since it just prints the type + "id" for the object
I'm an OS developer and in my view using printf is like seeing only half the world at once. There's a whole world of platform specific decisions that are made at compile time and runtime that you can only see through the lens of a good assembly debugger.
You're also talking about debugging apps running comfortably in the idillic world created by the OS. It's much harder to debug foundational pieces with printf's when the program immediately panics or early printing isn't available.
In my opinion it's good to build habits that can be generalized to all sorts of software and not limit oneself to writing code in a highly structured environment where most of the work is done for you. I can trace through a program faster than someone can insert/remove printfs and recompile their program, and I don't need to think about what to print. I can look at everything at that point in time, covert data to strings, look at the stack, registers, etc. Very powerful stuff.
From my (super limited) experience, debuggers shine when:
- You're using a dynamically typed language.
Something like Rust can eliminate most bugs that come from incorrect data types. For me, a lot of bugs used to come from types that were different from what I expect.
- It is super easy to run your program with a debugger attached.
If your code needs to run on a K8s cluster or a dedicated test machine instead of locally, logs are much easier to get hold of than a remote debug session. Some people aren't even aware that they can attach a debugger to a computer over the network, or inside a Docker container.
- Your environment.
If you don't use an IDE that supports a debugger, it's another friction point. I'm not sure if Vim has something similar to, say, PyCharm's debugger.
Similarly, if you're a junior, and you reach out to a senior and they tell you to debug using logs, you probably will never switch to using a debugger yourself.
If you use a time travel debugger (rr-project.org, undo.io - where I work, or Microsoft's WinTTD) you can generally just record an application to a file and debug it (with full fidelity) somewhere else.
And you don't need a full debugger setup on the target machine, just the recorder binary.
Gives you the possibility to have a proper debug experience without having to set up debugging that somehow works in a live k8s pod, or connects through special firewall holes or somesuch.
Inserting a breakpoint is just as easy as a printf, and as long as you're still using a debugging build, you don't have to recompile. With the printf you might not have considered all the variables you need, so you have to go back, insert, and recompile. With a breakpoint you can inspect the contents of anything at that scope, and even see what the code flow is with that given state. You can even save a core dump to go back to later.
You can also script breakpoints to output the info you want and continue, giving you your information ocean.
Basically, a debugger is a more efficient and powerful tool. In the one situation where you're not skilled with a debugger feature, a printf can be quicker than having to learn, but it's objectively worse.
You can insert and remove breakpoints while running. You can inspect variables the instant you realize they might be relevant.
During my long career, I’ve always been told “You should know you code well enough that a few well placed printfs is the most you’ll need to understand a bug”.
But, most of my career has been spent debugging large volumes of code written by other people. Code I’ve never seen before and usually will never see again.
A debugger making a 10X productivity difference for me is no joke.
> With the printf you might not have considered all the variables you need, so you have to go back, insert, and recompile
In some languages, such as Python, it's fairly easy to write a debug-print function that prints all the local variables (as well as the function name and line number it was called in).
That misses the mark. You can’t really compare a hackish ”print the world as a string“ function against a debuggers ability to stop time, walk around, pick things up, slice them open, put them somewhere else, and start time again.
That’s not just not the same league, it’s playing a whole different game.
You call it haskish but it's something I've done in the pasty and find useful.
I don't find debuggers all that useful, because I often find I'm spending more time thinking about how to use the debugger rather than how to fix the bug; since debugging is hard I want tools that I don't have to think about at all, as they distract me from thinking about the bug.
Maybe that's because I don't have enough experience with a particular tool. If I used a debugger more often it would come naturally to me. But I find most of my bugs are simple enough that that doesn't happen, because I write modular code and TDD.
I mean sure, logging metadata is better than nothing. But that’s akin to saying you don’t need a car because your feet have never failed you at crossing distances. Yes, you need to learn driving first, and that can seem hard at the beginning, but I doubt you’d want to go back to walking after.
To each their own, but I wholeheartedly recommend learning about debuggers. It should be one of the core tools of every software engineer.
You can trivially add printf-like statements on the fly with debuggers; with GDB, one method I often use is the ability to attach commands to breakpoints, to be executed when the breakpoint is hit.
break SourceFile.cpp:123
command
pp var1
pp var2
continue
end
With that `continue` at the end there, this breakpoint will not pause execution (except to run the commands).
> If you have a fairly shallow bug ... But most of the bugs that I see aren't that shallow
Oh come off it, debuggers shine the brightest when there are lots of unknown unknowns. With printf debugging you can peel back exactly one layer at a time (oops, need to log one more thing) whereas with a debugger you can slice through the Gordian knot.
Try using a debugger to debug a globally distributed system that humans aren't given access to for security reasons in a post mortem and then "come off it" yourself
I use profiling tools more than step-debugging, and printf()/var_dump()/IO.inspect/System.out.println/&c. much more than both, because most of the time I just need to see what the data looks like in a few locations to have a solution.
Sometimes the problem doesn't show up immediately in data and the code is too complex or uses a lot of wormhole techniques like particular forms of exception abuse, that's when I might fire up the debugger and browse frames instead.
I trend more towards print debugging than breakpoints.
To me the beauty of print debugging is you can see the flow, and see it quickly in contrast to the debugger. Simply with the debugger, a lot of the time is spent stepping past (at the moment) superfluous breakpoints.
Step, step, step, step, …, step, step, BANG!
Versus a quick BANG preceded by a trail of debris I can then postmortem. I use both, naturally, but prefer the crashes with a debris field than walking on eggs to potential disaster.
Which is why a good reversible debugger is so powerful. (I use rr-project.org and Pernosco but I imagine undo.io is similar.) Don't worry about stopping at the right point. Go back to the critical points over and over again, and go backwards to what generated an input. If you like logs, the debugger has facilities for generating tailored ones. Or manually generate your own debugging session log trail, generating entries throughout the execution in any order, and see them in execution order.
> I wonder what makes our experiences so different? Do you unit test a lot? Particularly with TDD? I am guessing that this practice means I just don't get to see a lot of the bugs that a debugger would help me with.
Debugger helps a lot when performing TDD. You will enjoy setting a breakpoint to check whether a line is hit as expected by a test case.
My grief with debuggers is due to C++ and template code (usually STL) and optimizations zeroing out values. I wish it had better training wheels to say "nope that's STL library code, you probably don't want to step any deeper". That's largely a criticism of C++ itself, though. But yes for this reason I prefer printfs despite 20 years in the game.
I think unit tests and debug/verbose print statements are useful for debugging. Even if you're firing up the debugger, these provide a lot of information up front! But more importantly, they make your project more approachable for contributors who aren't already familiar with the debugger.
I've also used both GUI-based debuggers and printf, and I prefer printf. But the most important thing it to write your code so there aren't many bugs and when there are they are easy to find. I do this using modular code, unit tests and regression tests.
But also, experienced programmers should never forget their printf debugging roots.
I was debugging something earlier this week that was hit like a hundred times in a tight loop. After the first dozen or so times I told gdb to continue, I realized, wait, this will be faster if I just fprintf some relevant information to a file. Sure enough the file pointed me in the right direction, and I was able to go back and get fancy with "disp" and "cond" and hit that breakpoint only when I needed to.
Essentially you set it like a breakpoint (attaching a printf style string to a code location) and then just "continue" until you've gathered what you want.
> What happens if the loop executes some non-idempotent calls? I guess printf debug still has some value :)
Then they'll do the same thing when you replay.
Non-idempotent system calls are tricky because they interact with the outside world - but that's still OK.
In Time Travel Debug, the process you're debugging is essentially in the Matrix. When it's being recorded everything acts as normal (and it'll see the real results of those non-idempotent calls).
When it's being debugged, any interaction with the outside system is prevented and replaced with the behaviour we saw at record time. It'll still think it's doing the non-idempotent calls, they just won't change (or depend upon) the state of the rest of the system.
Um, is it okay to admit, as an "experienced" programmer, that I often resort to print statements? I mean, compilers are just so darn fast these days.
Another trick: for rare circumstances, code whatever complicated logic is needed to isolate the bug in order to issue a print statement, then use the debugger to break on that print statement.
Sometimes I have had much better results with adding logs especially if an issue doesn't occur always and I am not so sure about the steps either because breakpoints take a lot of time as well. Also, in some cases (esp. mobile UI) breakpoints might actually break the flow and you might not get a proper flow. But yeah mastering the debugger is indeed a must and a GUI debugger is better than a CLI debugger. It's just that at least for me personally logging is first line of debugging :|
> Sure, it doesn't help much for some scenarios (one I've heard people mention is multithreaded code, where logs are better?), but for most people it's not that far from a superpower.
Debuggers can be great for understanding multithreaded code - and you can potentially freeze threads and continue others in order to provoke a particular race condition.
However they're potentially quite weak at stepping through a concurrency bug - stopping after each line to understand the sequence of events has a good chance of making your bug go away.
I'd say you want Time Travel Debugging if you need to capture and step through a rare event: you get to record the bug happening (without interrupting it) and then step through the recording.
On Linux, Undo.io (disclaimer: where I work) and rr (open source) are good at this.
I've had logging slow down a concurrency issue enough to cause the race condition to never appear when logged, but setting a breakpoint (and stopping all threads) prevailed.
I would add to that that in most scenarios where people think debugging doesn't help or won't work - it can.
Running inside Docker, multithreaded, multiprocessed, all can be debugged with a little effort. Most often much less effort that repeatedly printf debugging.
People who think that case X cannot be debugged without printf often don't know the features of their debugger. I.e. look at several of the comments which seem to miss that you can:
- Remote debug.
- Use conditional breakpoints.
- Use breakpoints to trigger commands, e.g. log values, enable other breakpoints, etc. instead of stopping. execution.
Just yesterday I gave a talk at MeetingC++ in Berlin on debugging multithreaded code. It's amazing how few developers know anything beyond the very basic of their debugger. If all you know is print, break, continue, next and then you dismiss the debugger as "not very useful" then you've not made a judgement based on information but on initial reaction.
Many people believe that a debugger won't work in their specific scenario, but often they are wrong; debugger can connect across network boundaries or into other processes that weren't launched by the IDE
Idk. I feel like the second you need to use a debugger your code and design has become too complicated and needs to be rethought.
In general anything you would want to debug should probably be exposed as a unit test and the area of concern should have test cases made that trigger the behavior you are concerned about.
The entire process of debugging essentially results in the same process as you would need to do to create unit test. While it is faster it is lost once done, making the entire process one shot.
Some ancient version of NetBeans leaking ram like a sieve until it brings down the machine, or a decade old version of Eclipse that can't pull in a newer CDT, running on a fork of OpenOCD with nothing customized for the CPU architecture running dog slow.
Sadly, it can be faster to reserve a GPIO, bitbang a TX-only UART, and get on with it.
Rider shows the value of assignments in your code, on the line they are assigned when you are debugging. This has saved me time when I notice that a value I didn't even think about looking into was wrong.
I use debug logs extensively. I log a LOT. I can put the logs and code next to each other and trace through the code. So much better than a debugger. With the logs, I don't have to worry about timers or concurrency or any of that. I can take my time and read the code and reason about what's going on.
Edit: Logging helps me look at what is going on in prod as well. I can trace messages/transactions completely through the path and if there's an issue, I'll see it.
I've been writing C code a lot more like scripting after familiarizing with how easy it is to use gdb compared to jumping through hoops say in vscode to setup a new build target for every new C file I want to mess with.
Just write `test_x.c`, `gcc -o test_x test_x.c -g`, and `gdb --args ./test_x --blah --blah` allows for much faster iteration on things.
PLUS the gdb commands end up far easier and more powerful to probe things than mess with the GUI most of the time.
I honestly believe it's a cultural thing. I don't think there's any rational reason to not want to use a debugger, but I've met many people who swear it's useless. I think people in web tend not to use debugger, and a lot of Linux people as well. Everybgame developer I've met and most windows programmers use them
I made heavy use of interactive debuggers earlier in my career. After several years working in environments where debuggers were broken, unhelpful, or not available, I completely lost the habit. It was not so much a cultural thing as simple practicality: logging always works. I'd rather maximize my limited brainpower by focusing on the software I'm building and thinking as little as possible about the tools I'm using.
It may be somewhat cultural in that influence from functional programming eventually changed the way I think about state and state transitions, leading me to design my code differently, reducing the amount of debugging I have to do and making it easier to do via logging.
Even in multithreaded code, it's absolutely amazing to be able to pause a running program and look at the list of running threads and the values in scope and see where deadlocks might be sitting.
It's immediately obvious you're deadlocked, which is actually kind of tricking to suss out with log-style debugging.
Modern debuggers can do so much, being able to lay down conditions to only break when certain values are set, etc. etc. Some can even "rewind" programs. I'd say most people (including myself) are using only 25% of their debugger's capabilities.
Aside: One the reasons I despise working with async Rust code is the mess it makes of working with a debugger.
Slightly off topic but I think it is a good place to ask:
One of the few things from windows that I miss when using linux is the debugging experience with visual studio (not code). When debugging a medium-sized C++ project on windows, the launch of the debug build is pretty fast and stepping over lines is almost instantaneous. On linux launching the executable using gdb takes like 10 seconds loading modules and stepping over each line takes like half a second which I think is intolerable (lldb is even worse). Yet I don't see people complaining about this online very much. Am I missing something? E.g. is there a compiler flag that speeds up debug launch time and step speed that I am not using?
Yea, our software stack pulls many dependencies (our software probably totals over 200k loc). We depend on Qt, OpenCASCADE, and a few other heavy C++ Libraries. Stepping over a single line of code in GDB using the TUI can take 3-5 seconds in the worst case. I’ve been meaning to investigate or profile it further when I get the time, but it functionally means I avoid using the debugger except as a “last resort”, or only using it to catch segfaults or unhandled exceptions.
It’s very odd. It’s like it doesn’t cache something and ends up doing some strange expensive symbol search every time it hits a breakpoint or something.
Curious if anyone has a good solution to this also
A long time ago, "set use-deprecated-index-sections on" improved some things; don't know if any modern compilers need that anymore. I also have a "disable pretty-printer global builtin" in my ~/.gdbinit, though I don't recall what that was for, or if I even determined that to improve speed. Currently it seems "set style sources off" reduces some startup time. Don't think I've debugged anything your scale though, and I doubt any of my suggestions will actually help.
Ensure you're building with DWARF5, and enable accelerator tables like .debug_names, which will allow debuggers to receive a pre-prepared index of the symbol names in the program (and thus it doesn't have to parse all the DWARF on startup).
Slow stepping is a surprise; there's no OS reason for that to be slower. Possibly if your types are really large and complicated, the debugger has to fetch a lot of data to refresh its view of state each time?
> I don't see people complaining about this online very much
I complain about gdb all the time, speed is just one aspect. Step-by-step debugging is just terrible on Linux. Maybe that's actually the reason few people complain about it, they just don't use gdb, instead relying on other tools, especially printf(). I am not in the video game industry, but they seem to be way, way ahead of everyone else, especially Linux (non-game) developers. Maybe some collaboration is in order.
As for your specific problem, I don't know. Do you have optimization turned on when debugging? gcc/gdb and the LLVM equivalents let you debug optimized builds, but it is not ideal as knowing which instruction corresponds to which line is complicated, and maybe gdb is working extra hard for it. The "-Og" flag is supposed to only do "debugger friendly" optimizations, also "-ggdb" or "-ggdb3" is supposed to be better than plain "-g" for use with gdb.
Haven't observed anything like that. Even with remote gdbserver on a low power embeddeded device, stepping has always been instant for me. Could be a C++ vs C thing.
The only thing which takes time is debuginfod downloads.
This is opposite the typical experience so you have some wedrd setup. Gdb is instant and VS' debugger takes forever. It's why companies like epic games have their own debuggers
I built it and tried it out a bit with Godot on Linux. It seems OK (the UI is a bit on the "- how many widgets do you want? - yes" side), but also a bit janky. Trying to change the font for the editor didn't work, hovering over a variable to see its value either does nothing (but there is a sub-second cursor change that indicates something is supposed to happen) or it shows an error from GDB about trying to use an expression with a type or keyword (so there was an intent to show a value on a tooltip, it is just broken) - doubleclicking on a variable does add it in some panel with its current value and a timestamp, so the functionality for reading values/expressions from the UI is there too, just not done in the same way as the tooltips.
If polished a bit it could be useful, though from all the frontends i've tried the one i disliked the least (none are great) is Gede[0] (which i just noticed had a new release a few hours ago) as it has a very simple and straightforward UI and while it doesn't expose much functionality, what exposes seem to work fine without bugs.
[0] https://gede.dexar.se/
Hi, Seergdb author here.
Thanks for your honest review. Can you explain more about not being able to set the editor font? My tests show that as working. Make sure to "Save Configuration..." to make things permanent.
Also, for the "show variable value on hover", I'll test it.
Can you (or others) create an issue on my github page for any bugs or feature requests?
Thanks.
I submitted a bug report for the font:
https://github.com/epasveer/seer/issues/265
Thanks. I see it. I also added a bug report for the janky "show variable value" on mouse over.
> it has a very simple and straightforward UI and while it doesn't expose much functionality, what exposes seem to work fine without bugs.
Nice one, I will add it to my notes to use it next time I need debugging. The least thing I want when looking for a bug in my own code is to have to deal with bugs in the debugging tools.
GDB also has a built-in text user interface (TUI) that is surprisingly easy to use[1]. It even supports mouse interaction.
[1] https://sourceware.org/gdb/current/onlinedocs/gdb.html/TUI.h...
Personally I prefer cli over tui, but you can just toss something like this in a `.gdbinit`:
Neovim + nvim-dap + nvim-dap-ui + gdb works much nicer than that.
Only works if GDB has been built with TUI support, sadly :(
… If you are debugging c code surely you’re able to compile a debugger with whatever options you want?
In theory, yes, but in practice, people who know how to compile gdb is a small subset of people who need to debug c code.
Not to mention that it actually takes extra time to do so, when people are used to debug Python/JavaScript/Go code with one single click these days.
Just grabbed the source, and it's a pretty bog standard ./configure ; make. Simple build instructions in the readme. Not sure this is a daunting challenge for any even moderately skilled developer.
Most people who use C nowadays are students, not "moderately skilled developers".
Students barely know how to manually put together a Makefile.
As long as the number of programmers is growing rapidly, it’s likely that most people who use any language are probably students. The distribution automatically skews toward beginner.
That has no bearing on whether we should have good tools for skilled developers, including debuggers & makefiles & source distributions. Eventually, students become moderately skilled. And we used gdb in class when I was a student.
> That has no bearing on whether we should have good tools for skilled developers, including debuggers & makefiles & source distributions.
Unfortunately, it's also the best explanation as for why we don't have them in practice, and why all software seems perpetually developed by fresh juniors (because it is).
I don’t quite understand where this is going nor why. This little sub-thread stemmed from a good tool that does already exist, and from unsubstantiated claims that C devs are students and that the simple build steps are too hard for most people. Sure, a lot of software is written by junior devs but that has no bearing on whether gdb exists, or even how hard it is to build. We have lots of good debuggers, and luckily it only takes a few experts to write them, which is why we do have them in practice.
> This little sub-thread stemmed from a good tool that does already exist, and from unsubstantiated claims that C devs are students and that the simple build steps are too hard for most people.
It's next to impossible to properly source any claim in this area; as for a lighter standard of substantiation, C wasn't a big deal in university courses 15 years ago, so it's unlikely to suddenly become it now.
N=1, but back then, between C++ and Java courses jumping straight to IDEs (to focus on language instead of build steps), and Unix/C course sticking to direct calls to GCC, at least my year at my uni managed to go through 5 years without much exposure to make and Makefiles...
Anyway,
> Sure, a lot of software is written by junior devs but that has no bearing on whether gdb exists, or even how hard it is to build.
But it does have a bearing on GDB evolution and GDB GUIs, which almost universally expose less than the bare minimum of useful GDB features; of those that expose more, I'm yet to find one that works.
> We have lots of good debuggers, and luckily it only takes a few experts to write them, which is why we do have them in practice.
Well, yes. WinDbg, that debugger in Visual Studio, etc. :). GDB, too.
My point here is that, for better or worse, size and type of the target audience determines how much and what kind of attention the project gets. "Skilled, experienced developers" are a small niche; the vast majority of developers are fresh juniors (per the growth argument), creating pressure to satisfy them at their level. Which, for GCC, I guess it means the build steps remain arcane even relative to other GNU projects, and powerful GUI frontends for it are not a thing.
> Most people who use C nowadays are students, not "moderately skilled developers".
That seems a very dubious claim, especially since most students begin with Python or JS. Do you have any data to back that up?
Where? How?
Last time I tried to build GDB from source, which was some two months ago, it wasn't in any way simple. GDB comes embedded in some GNU binutils repo, instructions to build it in isolation weren't obvious.
I ended up creating a new VM with a more recent Linux distro, that came with newer GDB, and migrating everything I'm working on to it, because that was much easier than building GDB from source.
The GDB source is available as a standalone package: https://www.sourceware.org/gdb/download/
I'm not sure where you've been looking, but source tarballs have been available from both gnu.org and the GDB website for over 20 years, with every major release going back 30 years available for download.
Is it not usually? I've never had to compile gdb myself to get TUI
This has definitely been an issue for me before, I think at least for a while Ubuntu/Debian didn't ship gdb with tui enabled, you had to build it yourself.
Only seen on some minor embeded systems but it’s not a big deal to compile gdb with tui support enabled.
How have I never heard of this before? Thank you.
After trying many frontends for gdb I find that the TUI is the best. You just need to know about Ctrl + L to redraw if your program is printing stuff because the interface then become garbled.
I just put :
in my $XDG_CONFIG_HOME/gdb/gdbinitI like a colored prompt with
and I save history withI like to do that as well. Just want to keep it short :) Also I use vim mode for bash and since it's in .inputrc I also have a vim mode in gdb which I like a lot even though it's as good as zsh's
I've used "gdb-dashboard" a lot, would recommend it. It's similar to the TUI (though I haven't used the TUI much), but you can pick and choose to display a large variety of information and the colors make the output much easier to read.
You can also make the dashboard display on another or across multiple terminals, letting you create a much nicer window layout. I've scripted this up with tmux before to have it automatically create the terminal layout and connect them to gdb, you can create really nice layouts that way (though it can be a lot of effort).
ctrl+l is a must know for TUI stuff, including when vim messes up. that fixed so many mysterious "crashes" for me after I found out about it.
gef has tmux support so program's output goes to other tmux pane
gud-gdb frontend inside Emacs is quite convenient and easy to use.
In case you are in Windows, and connected to Linux and/or using WSL, you can also use WinDBG/VisualStudio to debug (remotely) Linux processes!
As long as you have gdbserver running on the remote end, no?
This is usually not how wsl stuff is implemented. Usually applications set up the Windows to WSL communication themselves including Microsoft products. Might be different in this case of course but then the experience would be a lot less smooth than what Microsoft themselves set as the default with the VSCode remote extension.
I keyed off “Linux and/or wsl” specifically. I didn’t know wsl had that interoperability cooked in, I would have assumed the vm still needed gdbserver running in order to connect for a “remote” debug session.
This is a Qt UI for GDB.
There's also gdbgui that I know of, a web-based UI for GDB:
https://www.gdbgui.com/
Always good to see more movement in the debug tooling
I like how gdb with qt creator works without any complicated configuration. You click on some breakpoints then hit play and the ide takes care of the rest.
Qtcreator is phenomenal. I don’t like using anything else if possible.
VSCode's built in debug interface is also pretty great. I use it with the CodeLLDB extension and LLDB.
I don't really understand why people would use a separate debugger to the one that their IDE has. Most IDEs have solid debugger interfaces.
and drop down into gdb for complicated prints of variables and such
To add on to the pile of more GDB GUIs, here's one I've made: https://github.com/dzaima/grr. Though it's still got a fair bit of missing features that may be essential for some uses (due to my usage being largely assembly-level, which doesn't need too many fancy features).
And DDD. A Motif front end.
Vscode also has an ok gdb frontend, very nice when you are debugging embedded microcontrollers
Speaking of web-based debuggers, I recenty created a similar project but focused on x86-64 assembly debugging: https://github.com/robalb/x86-64-playground
A mediumish discussion 2 years ago https://news.ycombinator.com/item?id=33044885
For the Emacs users in the crowd, GUD is a pretty great GDB integration.
Ever since the advent of LSP, Emacs has felt superior to everything else. I have no reason to leave it. Especially once they made it faster with native comp.
Like why should I keep trying this month's new editor with a couple new gimmicky features, when I can just pop a plugin onto Emacs that adds that exact feature set, while maintaining everything else how I like it.
I first really got into coding when Atom was a thing, and then that died off and became VS Code and I was pretty sad about it, because while VS Code is good, it doesn't follow the same philosophy as Atom. But then I took the time to learn Emacs ~4 years ago, and nothing new ever comes close to convincing me it's outdated tech that I need to move on from.
That was a random rant, but I'm just really appreciate Emacs, and I'm glad it's stuck around.
Even before the dawn of the LSP era, Emacs was pretty great with ctags.
Yeah I first gave it a try before LSP and I don't think I was ready to be redpilled yet, because it didn't stick. So I can't comment on the state of it before then. I kinda joined once the LSP stuff got fairly smoothed out.
I prefer the GDB Graphical Interface in Emacs[1] (M-x gdb), rather than the more basic integration via GUD[2] (M-x gud-gdb). I’ve had to switch to GUD to run lldb recently, and I miss having dedicated windows that show breakpoints, threads, the current stack, etc.
The one nice thing about GUD is that the interface is consistent across debuggers, so I don’t need to refresh myself on the keyboard shortcuts when switching between debugging Python with pdb and C++ with lldb.
[1] https://www.gnu.org/software/emacs/manual/html_node/emacs/GD...
[2] https://www.gnu.org/software/emacs/manual/html_node/emacs/St...
dape ( https://github.com/svaante/dape# ) is a great option for languages with debuggers implementing the Debug Adapter Protocol. Combined with debugpy it's killed M-x pdb for me, and has a very similar UI to M-x gdb.
lsp-mode+dap-mode also work well but requires some manual fiddling with launch.json files
oh nice - reminds me of DDD(1) DDD was like magic first time I saw it. Oh wow - DDD is still maintained ?? :-D
1 https://en.wikipedia.org/wiki/Data_Display_Debugger
2 https://www.gnu.org/software/ddd/
DDD was taught to me when in University, 20 years ago, and it already felt clunky, my views are now much more moderate but Motif still feels like an eyesore.
Conversations over the years have shown me that DDD was a great inverse marketing tool, ironically pushing developers towards the embedded debugger UI in their favorite IDEs... despite DDD itself being indeed very powerful. But even "usefulness over aesthetics" has its limits!
There's one DDD feature that I haven't found elsewhere: its graphical representation of a struct and its contents. You can double-click on a pointer field and then it draws whatever that field pointed to, with a nice arrow connecting the two.
I've found it a very powerful yet compact way to visualize the state of a program when debugging.
yes! this was so great in college to learn pointers and visualize linked lists
Reminds of SmallTalk / and Lisp's - https://en.wikipedia.org/wiki/Common_Lisp_Interface_Manager
There was even a story, that (at least for Common Lisp), you can start from almost blank state, but have an exception handler installed (that can continue), so as you go you live-edit and add pieces missing, or if code crashes change.
This is all good, until nowadays, where you really want to know what's deployed in production, and not just the last stuff I've live fixed.
I mean, I guess both have values tbh, but hard to pull two models like this and use... bit like - debugger or printf statements (or both!)
Current DDD under the updated OpenMotif with TTF fonts can look much better than it did in the 90's and 00's, miles ahead than LessTif/former propietary Motif. It blends perfectly with EMWM where I have Liberation Sans/Mono for almost everything.
motif does not work well with high resolution display, sadly
Motif today supportṡ TTF and for the HD issues, you can the DPI option for X11 at ~/.Xdefaults
Citation needed. I use Motif programs without issues on high resolution displays.
Yeah, I remember DDD being an incredible tool back in the day, but it was clunky even when it was new (1995).
https://www.gnu.org/software/ddd/manual/pdf/ddd-paper.pdf
> DDD is still maintained?
Absolutely. I wrote about its features here https://begriffs.com/posts/2022-07-17-debugging-gdb-ddd.html
Since the article was written, the maintainers fixed the issues I pointed out. No need for many of those workarounds now. Versions 3.4.0 and 3.4.1 are substantial.
Is there any URL that lists what's new in v3.4.0 and v3.4.1?
The DDD website ( https://www.gnu.org/software/ddd/ ) points to the source tar.gz and the full manual, but nothing that says "What's New" in recent versions.
Might not be as direct as you'd like, but all well managed¹ GNU projects ship a NEWS file in their tarball. In this case you can also read it from savannah² without fetching the tarball.
¹ https://www.gnu.org/prep/standards/standards.html#NEWS-File
² https://svn.savannah.gnu.org/viewvc/ddd/trunk/doc/NEWS?view=...
For me it reminds me of Insight, which was my favorite GDB front-end and worked great on Windows back in the cygwin era. Unfortunately it's definitely no longer maintained. https://sourceware.org/insight/screenshots.php (Well, I guess someone did rescue it to put it on github and do some work: https://github.com/antony-jr/insight )
I love that DDD has a variety of graphical visualisations built in. I always thought the ability to data structures was particularly cool.
A while ago there was a project to port it to GTK3 but I think that went away. I'm glad the mainline project is still going.
yeah - the 'data display' part was the real killer feature :-)
DDD is great. I still use it, but I am a fossil. I sought out DDD when I was looking for something similar to dbxtool, which I used on the early Sun Microsystems machines. Folks today are spoiled with things such as Source Level Debugging.
And it still uses Motif! Awesome!
Coming from VMS at the time, I was confused why there was no decent full screen interface to gdb. DDD was such a disappointment in this regard.
I always liked the concept with DDD, but I could never keep it from crashing more than the program I was trying to fix.
It is much better now.
The GNU project (and RMS) seem to get a lot of meme-ridicule, but GDB is a powerhouse. I've only had occasion to mess with it a little, but it seems to have had such a huge impact on people's development work over the years.
Genuine question what does gnu get ridiculed for?
The GNU/Linux copypasta thing which is all over reddit all the time, for example. Saying the software is "bloated" is another trope. Thirdly, a suggestion that it (GNU, free software) is all very impractical or "backward". I guess in general a feeling that it's not "cool" or "hip", maybe?
I don't know, it seems very common to me. Have you not seen anything like that?
People on Reddit have call bin utils and core utils bloated? I sort of assumed we all just took those for granted as rock solid and the reason that we can have a free Unix operating system. Huh. Maybe I'm wrong though. What subreddits are saying this? Again, genuine question, I think I must be getting old!
I'm not on Reddit these days, and haven't been in a year or so maybe, so couldn't show you anything more than what you could find with a quick search on maybe r/linux or similar heavily populated subreddits related to these things.
I saw this comment yesterday and thought of this discussion though, maybe it helps illustrate what I was trying to say? https://news.ycombinator.com/item?id=42170536
The point being made in the comment linked to, briefly, is how non-SW engineers "turn their nose up at open-source solutions". It's a bit broader than what I was saying, but was roughly the kind of trend I was trying to allude to.
I must be checking the wrong forums. Never seen it.
Ok, that's interesting to hear. More reason to spend (even) less time on Reddit and Youtube, I suppose, which is probably where I've been over-exposed to these sorts of points.
When I used to program in C++ on Linux 10+ years ago, I used Qt Creator which has a built-in debugger (GDB frontend). It worked great and I don't see a reason to use anything else for C++ [and Qt].
I sometimes need to use gdb to investigate bugs in C or Ada, but it is not my main activity. As a result I will not invest days to setup a debugging environment that I will not remember how to use 6 month later. My solution: I use emacs and have a short note with instructions: M-x gdb -i=mi exe_full_name -p 29123 M-x gdb-many-windows set follow-fork-mode child
Anyone has experience with this and can compare it to kdbg?
This combo works well for me: neovim + nvim-dap + nvim-dap-ui + gdb (with native DAP protocol support).
Debugger vs printf:
Has anyone found a reliable way to use a debugger when you have a) multi-process b) multi-threaded c) async d) timeouts? I would love to use a debugger but printf and logs “just work”
The syntax is clunky, but watchpoints can do what you want.
https://ftp.gnu.org/old-gnu/Manuals/gdb/html_mono/gdb.html#S...
I used GDB to debug contributions to the ION Delay-Tolerant-Networking stack, which is composed of dozens of processes. After some practice it became super easy to attach GDB instances to several running processes and watch the C structs be passed between them.
GDB can do a, b, c (async Code is not really special?). d is going to be tricky and really depends.
Thanks to all the replies… ok, looks like I’m going to have another go at debugging!
Qt GUI itself can use gdb/ldb and display a fair amount of data structures from the standard library.
Note that gdb is also scriptable with python, so you can easily register your own printers.
See also https://github.com/nakst/gf
Is this using vscode's DAP under the hood?
No it is not.
It's using gdb's "mi" mode.
Long time gdb TUI user, migrated to VSCode recently. Haven't looked back. Occasionally I do use TUI to look at the disassembly.
eclipse-cdt includes a GDB integrated debugger UI for C and C++ since forever. What's new here?
Eclipse CDT is a whole IDE; is rather complex, and one might even say convoluted; and is Java based. Seer is none of these things.
Of course - it's not like a GDB GUI is a novelty in itself, there are quite a few. But a GDB-GUI-only utility is a meaningful and important niche to consider.
I've used classic eclipse-cdt for over a decade until it cannot keep up with post-C++14 standards. It seems that it recently regains activity with the introduction of https://github.com/ghentschke/eclipse-cdt-lsp
pwndbg, gef, edb
On a general note, I would recommend any new (and experienced!) programmers to master the debugging tools of their ecosystem. I've seen countless experienced developers use printf-based debugging and waste hourse debugging something which could've been easily figured out by setting a breakpoint and stepping through your code. This is also a good way to understand code you're unfamiliar with.
This is one area where I believe a GUI tool is so much better: I can hover over variable names to view their values, expand and collapse parts of a nested structure, edit values easily, and follow execution in the same environment I write my code in.
Sure, it doesn't help much for some scenarios (one I've heard people mention is multithreaded code, where logs are better?), but for most people it's not that far from a superpower.
Interesting.
My experience is the opposite: I see developers waste hours stepping through their code a line at a time when a few judiciously placed logs (printfs() are fine, but we can do better) would have told them exactly what they needed in a jiffy.
If you have a fairly shallow bug, that is a single point in your code that always behaves incorrectly, then I find debuggers reasonably effective.
But most of the bugs that I see aren't that shallow, with code misbehaving when the context is just so and perfectly fine otherwise. In those cases, I need to see lots of different invocations and their context. The debugger is like trying to drink the information ocean I need through a straw. A mostly plugged straw.
I wonder what makes our experiences so different? Do you unit test a lot? Particularly with TDD? I am guessing that this practice means I just don't get to see a lot of the bugs that a debugger would help me with.
(And it doesn't mean I never fire up the debugger. But it is fairly rare).
I have more or less the same experience like you. Logging is a very resilient and adaptable technique – I can use it on my laptop or on remote HPC clusters, almost regardless of programming language (except maybe Haskell), it works fine on parallelized code, and so on, with very little configuration needed. It’s also important to me that it can be done “async”, since some of my larger codes can only be run on HPC clusters by putting a job in a process queue and waiting.
I’ve tried debuggers and see the appeal but I find it less useful than print debugging / logging.
I also rely heavily on unit tests when writing new code, so that also reduces the surface that I need to look for bugs based on the log. Moreover, most of my projects have 1-3 programmers and can largely “fit in my head” (<10,000 lines of code), so it’s probably different if you work at a FAANG company or something.
I think you have a great point here. Debugging tools make you dependent on a particular environment. Printing based debugging can work pretty much everywhere. If you master printf programming you can solve any debugging task.
Yes, portability and simplicity are the best parts of printf.
> If you master printf
The skill ceiling is low. Printf only does so much.
You could rope in environmental optimization to the skill discussion -- the ability to isolate areas of functionality, replicate problems, reason about unknown state, and do the legwork so that you can quickly spin the increased amount of iteration required by a simpler debugging tool -- but by then you have thoroughly sacrificed both simplicity and portability and are far past the skill floor of a debugger.
If we assess this by looking for problems created by overcommitting to one approach or another, overcommitting to a debugger looks like burning time trying to get tooling to work on a problem that doesn't really need it while overcommitting to printf looks like spending way too much time iterating on tiny steps that could have been jumped over given better visibility. I've seen both, of course, but I tend to see more of the latter and more denial about the latter. When you're burning time fighting tools it's obvious. When you're burning time because you don't know how a tool could have saved you time, it's less obvious.
YMMV.
> the ability to isolate areas of functionality
This is the key. You need to be able to narrow down where the bug is.
Not the OP but...
> programmers to master the debugging tools of their ecosystem. I've seen countless experienced developers use printf-based debugging and waste hourse debugging something which could've been easily figured out by setting a breakpoint and stepping through your code.
If you're wasting hours with printf-based debugging, I don't think you've 'mastered the debugging tools of the ecosystem'.
There are multiple ways to debug - step debugger tools, printf, logging to a file, etc. Each have their place.
If you're spending hours on any one approach, and perhaps that's the only approach you know, that's a red flag.
If you've spent hours going through printf, logging and step debugging and STILL don't have a good answer... bring in external eyes.
I've found/fixed bugs in a few minutes because of adding some log stuff first, because in those cases, it's the easiest approach. In other cases, running a debugger and setting a couple breakpoints is indeed the easier approach to start with, and I've done that.
Sometimes you find it with the first approach, sometimes you need to try the next approach.
Why must this be a mutually-exclusive situation?
You can have the source code debugger log messages to the output window without having to add logging statements and recompile the affected code.
This is the 21st century.
see visual studio's tracepoint functionality - works in native and .Net languages, https://devblogs.microsoft.com/visualstudio/tracepoints/
sure, if you want the logging messages available for perusal when deployed in production, then this won't help.
Even better - use Hot Reload and tweak your code in the debugger - https://learn.microsoft.com/en-us/visualstudio/debugger/hot-...
[edit] GDB's equivalent to tracepoint is mentioned elsewhere in this thread - https://news.ycombinator.com/item?id=42147372
I would guess longer compile would encourage breakpoints over printf and this would be programming language specific.
Being able to change breakpoints at runtime helps a lot when tracking down something more complex. Visual Studio breakpoints are great, and they’ve added conditional breakpoints which are even better. Previously I would approximate this by having code specifically branch to hit a breakpoint, ‘if (X) { breakHere();}’
I write a fair amount of native C++ code but only call it from either Python or dotnet so when I make a mistake here it’s usually a segfault / memory access issue which kills the process. There might be a way to debug the C++ from dotnet or Python but logging to std out helps me isolate the location of the issue which is sufficient. It’s not a big enough problem and I worry that either writing tests in C++ or learning a native debugger will pay off in time saved.
The big ones for me with log/printf debugging are:
- I can get a good idea of the temporal behavior of the program, and I can just scroll up to see earlier state, rather than having to restart the program in the debugger. (I know that "time travel" debuggers exist, but I've found them finicky.) I can scrub back and forth through time just by scrolling.
- I can compare runs by diffing the logs. Sometimes that alone is enough to show where things start going amiss. Or I can keep instrumented logs from baseline runs.
- If there's a personally useful set of printf statements in an area that I'm in a lot, I can save those off to a patch file or a local branch. I don't have to reapply my breakpoints / watchpoints in the debugger each time. Easy persistence.
(That said, I do like to start with a debugger when tackling reproducible crashes.)
A good debugger can provide more than just stepping thru code.
In IntelliJ with Java, you can set conditonal breakpoints with complex evaluations, you can set filters (only hit a breakpoint depending from where it is being called), use exception methods that only hit on certain exceptions instead of a specific line code, you can also use logging breakpoints, that act like printf debuging, but you don't need to scatter your code with print statements all over the place.
You can group, add descripitons, disable, enable and add temporary breakpoints, they are pretty powerful! I just wish intellij had a time travel debbuger like Visual Studio Pro.
https://www.jetbrains.com/help/idea/2024.3/using-breakpoints...
> I just wish intellij had a time travel debbuger like Visual Studio Pro.
You might find our Java product interesting, it adds Time Travel Debug to IntelliJ - https://undo.io/products/java/
Undo captures everything the process does, below the JVM level, so you can reproduce / rewind any problem you record as many times as you want (and copy the recording out of production onto a dev machine to debug, etc etc).
Please get in touch if you'd like a free trial.
I agree with both of you. Printf is not enough, breakpoints are not enough. The solution lies between. Ability to gather rapidly relevant information to converge on wrong states.
ps: I wish I could work on a porcelain layer to manage the breakpoints in a more logical manner. Considering a problem you'd like to create different sets of breakpoints, run various tests and gather results to reviews. With the ability to add or remove layers rapidly. It's probably not too hard to do.
I have found combining these things to be useful: breakpoints that print stuff and auto-resume the program. Allows you to attach trace points at-will without requiring a recompile or losing state.
Yes losing state is a killer
Yes exactly, and I'll probably say very generally that I usually use breakpoints when I am in the exploration stage of a significant state bug, and I'll usually use logging when I generally know where the bug should be but I need to pin point the exact place
I've been, and still am, at the mercy of both of printf-debugging style and real debugger.
Long time ago, worked on a port of game from PC to Playstation 1. Since we had the Yaroze "devkit" (not really a devkit, rather amateur kit for doing games), printf debugging was the only thing available.
Things kind of worked, but when we #ifdef-out the printfs it was crashing (and no debugger). We somehow discovered that one of "printf" side effect was clearing the math errors.
Interesting. I had a similar experience: once the debugging instrumentation with #ifdef macros was switched off, the code that worked before suddenly crashed. In my situation it had to do with the stack, because my debugging macros used some local/"automatic" variables, and that had concealed the bug before in the DEBUG build.
One thing I also noticed is that using "problem-oriented" languages like Python or Java changes where you spend your time trouble-shooting: ironically, not where the problem is (business logic) anymore, those parts of the code indeed tend to work better, but intead you waste time with libraries (Java: CLASSPATH, Python: packages, all:version conflicts). In Contrast, in C/C++ it was mostly memory management errors and bugs in the actual business logic (the former is also a great distraction, somewhat diminished by the introduction of smart pointers).
At Google, had to do Java on borg, and used few times the "Cloud Trace" debugger (not sure how it was called), but it allowed you to watch multiple instances of your binary in production, and then add some isolated set of java statements around code blocks, this way you can say (somehow) - if you end up on this line, then "inject" (somehow) something to log out... and then you can add whatever you want to be logged (like arguments, variables around, etc.).
But then later it got scrapped, or something like it.
Cloud "debugging" when you have multiple instances is one of the cases where there is no suitable enough debugger (yet).
It depends on the code as much as anything. I wrote a regex engine in Zig, and the instant I get a bug report I set breakpoints on a failing test and step through.
On the other hand, I'm working on an interactive application, and when I see a problem with it, I add more logging statements until I figure out what the problem is. Any time the logs have excessive detail as a consequence, I gate them behind an 'extra' flag on a per-unit basis, only removing the ones which amount to "got here".
If I had to pick one technique, it would be logs. I naturally think in terms of a trace through the execution pathway, rather than a step-by-step examination of the state of a small window into the code. It clearly works the other way around for some people.
One thing that makes this approach better for me is that debug logging is literally free, Zig uses a lazy compilation model so logging code which doesn't apply to a release compilation isn't even analyzed, let alone compiled, let alone included. In a language which doesn't work that way, there's motive to use printf-only debugging, and clean up after yourself afterwards, and that's extra work compared to firing up a debugger. So it shifts the balance.
Another thing to consider and is important to me - logging objects and state isn't always so simple. It can often be easier for me to open the debugger to look at the state of an object which cannot easily be printed.
Out of interest - what sort of objects are hard to print in this way but easy to view in a debugger?
An object with many fields (in a language with no conveniences for it). An object tree with multiple levels of nesting. A list or dictionary of such objects.
In general, print-based debugging requires a greater degree of specificity. If you know exactly what you're looking for it's great.
If you are performing a more exploratory sort of debugging, a decent graphical debugger will save you a ton of time.
In my case it's basically everything since I work in java, jackson's object mapper can easily get stuck or deserialise something incorrectly if the class hasn't been annotated correctly. So it's simpler for me to pull up the debugger and I can see the "actual" data thats making up my object, it also lets me run "queries" against anything of interest (i.e. computed fields).
The default toString method I've found to be useless almost every time I wanted to inspect an object in our codebase since it just prints the type + "id" for the object
I'm an OS developer and in my view using printf is like seeing only half the world at once. There's a whole world of platform specific decisions that are made at compile time and runtime that you can only see through the lens of a good assembly debugger.
You're also talking about debugging apps running comfortably in the idillic world created by the OS. It's much harder to debug foundational pieces with printf's when the program immediately panics or early printing isn't available.
In my opinion it's good to build habits that can be generalized to all sorts of software and not limit oneself to writing code in a highly structured environment where most of the work is done for you. I can trace through a program faster than someone can insert/remove printfs and recompile their program, and I don't need to think about what to print. I can look at everything at that point in time, covert data to strings, look at the stack, registers, etc. Very powerful stuff.
From my (super limited) experience, debuggers shine when:
- You're using a dynamically typed language.
Something like Rust can eliminate most bugs that come from incorrect data types. For me, a lot of bugs used to come from types that were different from what I expect.
- It is super easy to run your program with a debugger attached.
If your code needs to run on a K8s cluster or a dedicated test machine instead of locally, logs are much easier to get hold of than a remote debug session. Some people aren't even aware that they can attach a debugger to a computer over the network, or inside a Docker container.
- Your environment.
If you don't use an IDE that supports a debugger, it's another friction point. I'm not sure if Vim has something similar to, say, PyCharm's debugger.
Similarly, if you're a junior, and you reach out to a senior and they tell you to debug using logs, you probably will never switch to using a debugger yourself.
If you use a time travel debugger (rr-project.org, undo.io - where I work, or Microsoft's WinTTD) you can generally just record an application to a file and debug it (with full fidelity) somewhere else.
And you don't need a full debugger setup on the target machine, just the recorder binary.
Gives you the possibility to have a proper debug experience without having to set up debugging that somehow works in a live k8s pod, or connects through special firewall holes or somesuch.
Inserting a breakpoint is just as easy as a printf, and as long as you're still using a debugging build, you don't have to recompile. With the printf you might not have considered all the variables you need, so you have to go back, insert, and recompile. With a breakpoint you can inspect the contents of anything at that scope, and even see what the code flow is with that given state. You can even save a core dump to go back to later.
You can also script breakpoints to output the info you want and continue, giving you your information ocean.
Basically, a debugger is a more efficient and powerful tool. In the one situation where you're not skilled with a debugger feature, a printf can be quicker than having to learn, but it's objectively worse.
You can insert and remove breakpoints while running. You can inspect variables the instant you realize they might be relevant.
During my long career, I’ve always been told “You should know you code well enough that a few well placed printfs is the most you’ll need to understand a bug”.
But, most of my career has been spent debugging large volumes of code written by other people. Code I’ve never seen before and usually will never see again.
A debugger making a 10X productivity difference for me is no joke.
> With the printf you might not have considered all the variables you need, so you have to go back, insert, and recompile
In some languages, such as Python, it's fairly easy to write a debug-print function that prints all the local variables (as well as the function name and line number it was called in).
That misses the mark. You can’t really compare a hackish ”print the world as a string“ function against a debuggers ability to stop time, walk around, pick things up, slice them open, put them somewhere else, and start time again.
That’s not just not the same league, it’s playing a whole different game.
You call it haskish but it's something I've done in the pasty and find useful.
I don't find debuggers all that useful, because I often find I'm spending more time thinking about how to use the debugger rather than how to fix the bug; since debugging is hard I want tools that I don't have to think about at all, as they distract me from thinking about the bug.
Maybe that's because I don't have enough experience with a particular tool. If I used a debugger more often it would come naturally to me. But I find most of my bugs are simple enough that that doesn't happen, because I write modular code and TDD.
I mean sure, logging metadata is better than nothing. But that’s akin to saying you don’t need a car because your feet have never failed you at crossing distances. Yes, you need to learn driving first, and that can seem hard at the beginning, but I doubt you’d want to go back to walking after.
To each their own, but I wholeheartedly recommend learning about debuggers. It should be one of the core tools of every software engineer.
You can trivially add printf-like statements on the fly with debuggers; with GDB, one method I often use is the ability to attach commands to breakpoints, to be executed when the breakpoint is hit.
With that `continue` at the end there, this breakpoint will not pause execution (except to run the commands).> If you have a fairly shallow bug ... But most of the bugs that I see aren't that shallow
Oh come off it, debuggers shine the brightest when there are lots of unknown unknowns. With printf debugging you can peel back exactly one layer at a time (oops, need to log one more thing) whereas with a debugger you can slice through the Gordian knot.
Try using a debugger to debug a globally distributed system that humans aren't given access to for security reasons in a post mortem and then "come off it" yourself
Is that the only kind of difficulty you can think of? Lol.
I agree with this comment but you really didn't need the first 4 words.
I use profiling tools more than step-debugging, and printf()/var_dump()/IO.inspect/System.out.println/&c. much more than both, because most of the time I just need to see what the data looks like in a few locations to have a solution.
Sometimes the problem doesn't show up immediately in data and the code is too complex or uses a lot of wormhole techniques like particular forms of exception abuse, that's when I might fire up the debugger and browse frames instead.
I trend more towards print debugging than breakpoints.
To me the beauty of print debugging is you can see the flow, and see it quickly in contrast to the debugger. Simply with the debugger, a lot of the time is spent stepping past (at the moment) superfluous breakpoints.
Step, step, step, step, …, step, step, BANG!
Versus a quick BANG preceded by a trail of debris I can then postmortem. I use both, naturally, but prefer the crashes with a debris field than walking on eggs to potential disaster.
Which is why a good reversible debugger is so powerful. (I use rr-project.org and Pernosco but I imagine undo.io is similar.) Don't worry about stopping at the right point. Go back to the critical points over and over again, and go backwards to what generated an input. If you like logs, the debugger has facilities for generating tailored ones. Or manually generate your own debugging session log trail, generating entries throughout the execution in any order, and see them in execution order.
> I wonder what makes our experiences so different? Do you unit test a lot? Particularly with TDD? I am guessing that this practice means I just don't get to see a lot of the bugs that a debugger would help me with.
Debugger helps a lot when performing TDD. You will enjoy setting a breakpoint to check whether a line is hit as expected by a test case.
My grief with debuggers is due to C++ and template code (usually STL) and optimizations zeroing out values. I wish it had better training wheels to say "nope that's STL library code, you probably don't want to step any deeper". That's largely a criticism of C++ itself, though. But yes for this reason I prefer printfs despite 20 years in the game.
Visual Studio has natstepfilter
https://github.com/ocornut/imgui/blob/master/misc/debuggers/...
gdb has a skip function and you can exclude STL headers and functions named in special ways so that reduces the noise quite a bit.
I think unit tests and debug/verbose print statements are useful for debugging. Even if you're firing up the debugger, these provide a lot of information up front! But more importantly, they make your project more approachable for contributors who aren't already familiar with the debugger.
Debuggers can do printf debugging too with tracepoints.
I've also used both GUI-based debuggers and printf, and I prefer printf. But the most important thing it to write your code so there aren't many bugs and when there are they are easy to find. I do this using modular code, unit tests and regression tests.
But also, experienced programmers should never forget their printf debugging roots.
I was debugging something earlier this week that was hit like a hundred times in a tight loop. After the first dozen or so times I told gdb to continue, I realized, wait, this will be faster if I just fprintf some relevant information to a file. Sure enough the file pointed me in the right direction, and I was able to go back and get fancy with "disp" and "cond" and hit that breakpoint only when I needed to.
You could also use GDB's Dynamic Printf (https://sourceware.org/gdb/current/onlinedocs/gdb.html/Dynam...) to do the logging directly from GDB.
Essentially you set it like a breakpoint (attaching a printf style string to a code location) and then just "continue" until you've gathered what you want.
Oh sweet. I didn't know about that. I will be adding that to my toolbox.
In Jetbrains you can also do a Conditional breakpoint to active only if i=100 or something.
Well yeah, but I didn't know I wanted i = 100 until I had examined the fprintf output.
You need a time travelling debugger!
https://undo.io/
https://rr-project.org/
https://learn.microsoft.com/en-us/windows-hardware/drivers/d...
What happens if the loop executes some non-idempotent calls? I guess printf debug still has some value :)
> What happens if the loop executes some non-idempotent calls? I guess printf debug still has some value :)
Then they'll do the same thing when you replay.
Non-idempotent system calls are tricky because they interact with the outside world - but that's still OK.
In Time Travel Debug, the process you're debugging is essentially in the Matrix. When it's being recorded everything acts as normal (and it'll see the real results of those non-idempotent calls).
When it's being debugged, any interaction with the outside system is prevented and replaced with the behaviour we saw at record time. It'll still think it's doing the non-idempotent calls, they just won't change (or depend upon) the state of the rest of the system.
Um, is it okay to admit, as an "experienced" programmer, that I often resort to print statements? I mean, compilers are just so darn fast these days.
Another trick: for rare circumstances, code whatever complicated logic is needed to isolate the bug in order to issue a print statement, then use the debugger to break on that print statement.
Sometimes I have had much better results with adding logs especially if an issue doesn't occur always and I am not so sure about the steps either because breakpoints take a lot of time as well. Also, in some cases (esp. mobile UI) breakpoints might actually break the flow and you might not get a proper flow. But yeah mastering the debugger is indeed a must and a GUI debugger is better than a CLI debugger. It's just that at least for me personally logging is first line of debugging :|
> Sure, it doesn't help much for some scenarios (one I've heard people mention is multithreaded code, where logs are better?), but for most people it's not that far from a superpower.
Debuggers can be great for understanding multithreaded code - and you can potentially freeze threads and continue others in order to provoke a particular race condition.
However they're potentially quite weak at stepping through a concurrency bug - stopping after each line to understand the sequence of events has a good chance of making your bug go away.
I'd say you want Time Travel Debugging if you need to capture and step through a rare event: you get to record the bug happening (without interrupting it) and then step through the recording.
On Linux, Undo.io (disclaimer: where I work) and rr (open source) are good at this.
On Windows, you have Microsoft's own Time Travel Debug solution: https://learn.microsoft.com/en-us/windows-hardware/drivers/d...
(nb. there's also GDB's built-in process record technology but I'd recommend against that for any non-trivial software as the overheads are very high)
I've had logging slow down a concurrency issue enough to cause the race condition to never appear when logged, but setting a breakpoint (and stopping all threads) prevailed.
I would add to that that in most scenarios where people think debugging doesn't help or won't work - it can.
Running inside Docker, multithreaded, multiprocessed, all can be debugged with a little effort. Most often much less effort that repeatedly printf debugging.
I am not sure I understand your point. Can you be more explicit ?
People who think that case X cannot be debugged without printf often don't know the features of their debugger. I.e. look at several of the comments which seem to miss that you can:
- Remote debug.
- Use conditional breakpoints.
- Use breakpoints to trigger commands, e.g. log values, enable other breakpoints, etc. instead of stopping. execution.
- Debug multi-threaded code.
- Disassemble a fragment.
I have a bunch of (36 if you're counting :) short videos and blog posts introducing the advanced features of GDB: https://undo.io/resources/gdb-watchpoint/
Just yesterday I gave a talk at MeetingC++ in Berlin on debugging multithreaded code. It's amazing how few developers know anything beyond the very basic of their debugger. If all you know is print, break, continue, next and then you dismiss the debugger as "not very useful" then you've not made a judgement based on information but on initial reaction.
Oh thanks, I know about that and agree. I did not understand the point about doing debugging in containers.
But I think I understand now. I guess they only debugging a program that is running in a container.
Many people believe that a debugger won't work in their specific scenario, but often they are wrong; debugger can connect across network boundaries or into other processes that weren't launched by the IDE
And a good debug log can help more than either of those.
Idk. I feel like the second you need to use a debugger your code and design has become too complicated and needs to be rethought.
In general anything you would want to debug should probably be exposed as a unit test and the area of concern should have test cases made that trigger the behavior you are concerned about.
The entire process of debugging essentially results in the same process as you would need to do to create unit test. While it is faster it is lost once done, making the entire process one shot.
At least in embedded, often the tools suck.
Some ancient version of NetBeans leaking ram like a sieve until it brings down the machine, or a decade old version of Eclipse that can't pull in a newer CDT, running on a fork of OpenOCD with nothing customized for the CPU architecture running dog slow.
Sadly, it can be faster to reserve a GPIO, bitbang a TX-only UART, and get on with it.
Rider shows the value of assignments in your code, on the line they are assigned when you are debugging. This has saved me time when I notice that a value I didn't even think about looking into was wrong.
I use debug logs extensively. I log a LOT. I can put the logs and code next to each other and trace through the code. So much better than a debugger. With the logs, I don't have to worry about timers or concurrency or any of that. I can take my time and read the code and reason about what's going on.
Edit: Logging helps me look at what is going on in prod as well. I can trace messages/transactions completely through the path and if there's an issue, I'll see it.
I've been writing C code a lot more like scripting after familiarizing with how easy it is to use gdb compared to jumping through hoops say in vscode to setup a new build target for every new C file I want to mess with.
Just write `test_x.c`, `gcc -o test_x test_x.c -g`, and `gdb --args ./test_x --blah --blah` allows for much faster iteration on things.
PLUS the gdb commands end up far easier and more powerful to probe things than mess with the GUI most of the time.
I honestly believe it's a cultural thing. I don't think there's any rational reason to not want to use a debugger, but I've met many people who swear it's useless. I think people in web tend not to use debugger, and a lot of Linux people as well. Everybgame developer I've met and most windows programmers use them
I made heavy use of interactive debuggers earlier in my career. After several years working in environments where debuggers were broken, unhelpful, or not available, I completely lost the habit. It was not so much a cultural thing as simple practicality: logging always works. I'd rather maximize my limited brainpower by focusing on the software I'm building and thinking as little as possible about the tools I'm using.
It may be somewhat cultural in that influence from functional programming eventually changed the way I think about state and state transitions, leading me to design my code differently, reducing the amount of debugging I have to do and making it easier to do via logging.
Even in multithreaded code, it's absolutely amazing to be able to pause a running program and look at the list of running threads and the values in scope and see where deadlocks might be sitting.
It's immediately obvious you're deadlocked, which is actually kind of tricking to suss out with log-style debugging.
Modern debuggers can do so much, being able to lay down conditions to only break when certain values are set, etc. etc. Some can even "rewind" programs. I'd say most people (including myself) are using only 25% of their debugger's capabilities.
Aside: One the reasons I despise working with async Rust code is the mess it makes of working with a debugger.
Slightly off topic but I think it is a good place to ask: One of the few things from windows that I miss when using linux is the debugging experience with visual studio (not code). When debugging a medium-sized C++ project on windows, the launch of the debug build is pretty fast and stepping over lines is almost instantaneous. On linux launching the executable using gdb takes like 10 seconds loading modules and stepping over each line takes like half a second which I think is intolerable (lldb is even worse). Yet I don't see people complaining about this online very much. Am I missing something? E.g. is there a compiler flag that speeds up debug launch time and step speed that I am not using?
Yea, our software stack pulls many dependencies (our software probably totals over 200k loc). We depend on Qt, OpenCASCADE, and a few other heavy C++ Libraries. Stepping over a single line of code in GDB using the TUI can take 3-5 seconds in the worst case. I’ve been meaning to investigate or profile it further when I get the time, but it functionally means I avoid using the debugger except as a “last resort”, or only using it to catch segfaults or unhandled exceptions.
It’s very odd. It’s like it doesn’t cache something and ends up doing some strange expensive symbol search every time it hits a breakpoint or something.
Curious if anyone has a good solution to this also
A long time ago, "set use-deprecated-index-sections on" improved some things; don't know if any modern compilers need that anymore. I also have a "disable pretty-printer global builtin" in my ~/.gdbinit, though I don't recall what that was for, or if I even determined that to improve speed. Currently it seems "set style sources off" reduces some startup time. Don't think I've debugged anything your scale though, and I doubt any of my suggestions will actually help.
Ensure you're building with DWARF5, and enable accelerator tables like .debug_names, which will allow debuggers to receive a pre-prepared index of the symbol names in the program (and thus it doesn't have to parse all the DWARF on startup).
Slow stepping is a surprise; there's no OS reason for that to be slower. Possibly if your types are really large and complicated, the debugger has to fetch a lot of data to refresh its view of state each time?
> I don't see people complaining about this online very much
I complain about gdb all the time, speed is just one aspect. Step-by-step debugging is just terrible on Linux. Maybe that's actually the reason few people complain about it, they just don't use gdb, instead relying on other tools, especially printf(). I am not in the video game industry, but they seem to be way, way ahead of everyone else, especially Linux (non-game) developers. Maybe some collaboration is in order.
As for your specific problem, I don't know. Do you have optimization turned on when debugging? gcc/gdb and the LLVM equivalents let you debug optimized builds, but it is not ideal as knowing which instruction corresponds to which line is complicated, and maybe gdb is working extra hard for it. The "-Og" flag is supposed to only do "debugger friendly" optimizations, also "-ggdb" or "-ggdb3" is supposed to be better than plain "-g" for use with gdb.
Try Jetbrains CLion
Haven't observed anything like that. Even with remote gdbserver on a low power embeddeded device, stepping has always been instant for me. Could be a C++ vs C thing.
The only thing which takes time is debuginfod downloads.
How large is your code base? I am talking about 100K+ LOC with many complex dependencies (mainly Qt modules).
2 Mio LOC, no problem here.
Sounds like maybe you have reverse-debugging enabled? I mean target record or target record-full?
Anyway, I seldom do step-by-step. I typically work with dprintf.
Are you talking heavily templated C++ code as well? I’ve got my suspicions about how much this affects things…
Nope, with reverse-debugging enabled it will be insanely slow, not something that you can miss.
I am curious does your project have large external dependencies or is it self-contained.
This is opposite the typical experience so you have some wedrd setup. Gdb is instant and VS' debugger takes forever. It's why companies like epic games have their own debuggers
Have you actually compared the debug performance of a large cross-platform application on windows vs linux?
I am not saying that you haven't just trying to make sure that your argument is backed by data rather than heresay.