tag:blogger.com,1999:blog-2425290326823263574.post5220594872870205121..comments2022-12-04T18:48:06.405-08:00Comments on Krazy Glew's Blog: Automatically Deducing Build Dependencies => Cross ToolRecursive Builds WorkAndy "Krazy" Glewhttp://www.blogger.com/profile/08442494949914217568noreply@blogger.comBlogger3125tag:blogger.com,1999:blog-2425290326823263574.post-27493560587032483602008-12-17T19:55:00.000-08:002008-12-17T19:55:00.000-08:00Andy, thanks for the plug. :-) I'm not a scons exp...Andy, thanks for the plug. :-) <BR/><BR/>I'm not a scons expert, but in my build system (called mbuild) I was very motivated by the expressiveness and the portability that scons derives from using a (relatively) portable scripting language. In this case uses python. <BR/><BR/>I didn't like the single scan-build ordering that is inherent to scons. It is possible to subvert that, but I found that mechanism unnatural and the idiom I found on their wiki would break as scons evolved through different versions. In the end, I missed the more obvious control that is afforded by systems like make.<BR/><BR/>For make-based builds, it was annoying to have to rely on different external scripting and tools systems on each platform. (ie. Do I force the use of cygwin on windows for things like "rm"?) <BR/><BR/>So I tried to make a hybrid. I cannot say that I solved the recursive build problem, but with mbuild, I do have a portable python based build system. I spend much less time tinkering with my build system than I ever did when I used make or scons. "build" became a nonissue for me and I can focus on other problems. For me, that makes mbuild a success. Now if I could only release it...<BR/><BR/>--<BR/><BR/>I've been tripped up by scons' default inability to deal with conditional includes. It is truly hard to know what you depend on for a particular build. I've been considering adding simple CPP "#if defined(yyy)" knowledge to mbuild's source dependence scanner, but that is a slippery slope.<BR/><BR/>--<BR/><BR/>A couple of years ago when Andy mentioned the strace idea, I briefly looked at using strace and found that there are truly an enormous number of files that get included or referenced in a typical C++ build. I was pretty surprised. I didn't want to have to list dependences on the "phase of the moon" as it were nor attempt to filter out which ones were really important and which ones were ignorable. And as the other commenters have pointed out, portability is an issue and filtering on different O/Ses seemed tricky.Mark Charneyhttps://www.blogger.com/profile/01010603302060197724noreply@blogger.comtag:blogger.com,1999:blog-2425290326823263574.post-18363939501031967002008-12-17T18:45:00.000-08:002008-12-17T18:45:00.000-08:00You read my mind. I have had this project in the ...You read my mind. I have had this project in the back of my head for years. Only drawback is portability. Every system is different.<BR/><BR/>On some systems you use strace, on Solaris dtrace might be the ticket. On Windows there is another solution, macos another. But you write that module once and you have an incredible build tool.<BR/><BR/>Basically you can build by example. You could even build it by hand once and from then on the tool could rebuild it as needed. And you get 'make clean' for free as well.<BR/><BR/>Also, the hooks used by stuff like Google Desktop Search to identify all files that have changed might work, if they can also flag files that are read.Waynehttps://www.blogger.com/profile/09009425518356678315noreply@blogger.comtag:blogger.com,1999:blog-2425290326823263574.post-9224681863537449262008-12-17T16:36:00.000-08:002008-12-17T16:36:00.000-08:00Andy: I think you're right on. The dependencies a...Andy: I think you're right on. The dependencies are the hard, magic bit - and in large software systems with modular structures and recursive build processes, getting accurate depdendencies is *hard* - so if you can get good deps the tool platform problems can be solved in lots of ways.<BR/><BR/>You have a couple questions in your article i'd like to address, since I work for Electric Cloud:<BR/><BR/><I>"Past conversations, either with Ousterhout or with EC sales folk,<BR/>lead me<BR/>(1) unsure as to whether they have done the truly automatic thing for dependencies<BR/>but<BR/>(2) sure that they are not "diversity tolerant"<BR/>- they are yet another tool that requires everyone, the whole project, to use their build tool."</I><BR/><BR/>1) Yep, we really have done the truly automatic thing for dependencies - ElectricAccelerator tracks at the file-system level what files are read and written, collecting the data required to build the dependency graph for a build (and also letting ElectricAccelerator know when a target was prematurely fired, in order to revert it and re-run once all implicit deps are satisfied). The approach Andy indicates earlier in his article - using strace to trap this info - is a great first pass; strace slows a build down 5x or more, though, and isn't available on Windows. ElectricAccelerator is built for speed, so we're more elegant about how we do the data collection.<BR/><BR/>2) ElectricAccelerator assumes as a commercial requirement that developers are tightly bound to their tools, whether by choice, edict, or (mis)fortune. A commercial product which requires code change <A HREF="http://www.codefast.com/" REL="nofollow">will fail</A>, so it's important to us to support specific tools. So we don't actually require everyone to use our tool - we emulate *your* tool in most cases - and compatibility with the original infrastructure is maintained.<BR/><BR/>If you send me a message at scas tle [at] electric- cloud.com, i'd be happy to send you John's paper or any of the other whitepapers we've produced.<BR/><BR/>- Scott CastleAnonymousnoreply@blogger.com