Optimization for reading *.d files

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Optimization for reading *.d files

This post has NOT been accepted by the mailing list yet.
Hi everybody,

I'm currently refactoring the build system at my company. We got around 20k C++ source files, and we use a form of (messed up) recursive make. After a lot of tinkering to get things right I got to 10 (maybe 5 with some more massage) seconds for a clean build. I need something around 1 second.

Anyway, I want to continue to use GNU Make, and not fallback to CMake/Ninja. After some profiling, what's killing me is parsing the "*.d" files generated by the compiler.

The time to include all dependency files of my project in one single makefile (as I want to get rid of recursive make), is 4 seconds.

I decided to get my hands dirty. I cloned GNU Make 4.0 and started analyzing the parser. Of course, the parser is complex. But those dependency files are so so simple! I want to leverage that information.

I added a detection so the parser knows it's parsing a ".d" file, and switches to my own "eval_d_file" function.

My function doesn't try to find special qualifiers, nor expand variables or anything like that. It parsers as if the only allowed syntax is what exists on what is generated by the compiler.

I got a speed-up. My original 4 seconds went to 2 seconds. Which awesome. Of course, I probably did a buggy implementation (since I don't understand everything going on the code), but it seemed to work.

There are lots of dependency files and they can be processed in parallel, before being merged into the database.

For that, GNU make could need an extension on the include directive to handle "include *.d" differently as it knows dependency files won't alter/create variables but just add dependencies.

I really don't have the guts to use my hackish make version in production, but I'm up to develop this functionality, if you agree it's a good proposal.

Breno G.