proposal: stop_at_err_count patch

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

proposal: stop_at_err_count patch

Richa Bindra
Hi,

Here is a small patch that adds a new option to make that causes make to abort after N failures. An extra argument --stop-at-err-count[=N] is provided which forces GNU Make to return immediately when the number of errors is reached. 
The current problem with this patch is that when GNU make exits, the children processes left are not killed by GNU make when it exits. I want all children to be killed before GNU Make returns. I am not well versed with GNU Make code, so any advice or feedback would be appreciated!

diff -u /home/foo/make_git/make/job.c make/make-4.2.1/job.c--- /home/foo/make_git/make/job.c 2018-10-04 16:30:01.108348000 -0400+++ make/make-4.2.1/job.c 2018-10-04 16:28:12.119052000 -0400@@ -478,9 +478,19 @@   const char *nm;   size_t l; +  if (max_stop_at_err_count <= stop_at_err_count)+  {+    error (NILF, 0, _("*** [%p] Error (aborted after %d failures) %d"),+        child, stop_at_err_count, exit_code);+    die (2);+  }+   if (ignored && silent_flag)     return; +/* Only increase STOP_AT_ERR_COUNT if the job _really_ failed.  */+  stop_at_err_count++;+   if (exit_sig && coredump)     dump = _(" (core dumped)"); @@ -595,8 +605,14 @@           /* We might block for a while, so let the user know why.              Only print this message once no matter how many jobs are left.  */           fflush (stdout);-          if (!printed)-            O (error, NILF, _("*** Waiting for unfinished jobs...."));+          if (!printed) {+             O (error, NILF, _("*** Waiting for unfinished jobs...."));+             if (max_stop_at_err_count <= stop_at_err_count)+               {+                 error (NILF,0, _("*** Aborting after %d failures"),stop_at_err_count);+                 die (2);+               }+          }           printed = 1;         } diff -u /home/foo/make_git/make/main.c make/make-4.2.1/main.c--- /home/foo/make_git/make/main.c 2018-10-04 16:30:01.120391000 -0400+++ make/make-4.2.1/main.c 2018-10-04 16:29:19.185057000 -0400@@ -216,9 +216,13 @@  /* Nonzero means keep going even if remaking some file fails (-k).  */ -int keep_going_flag;+int keep_going_flag = 0; int default_keep_going_flag = 0; +int stop_at_err_count = 0;+unsigned int max_stop_at_err_count = -1;+static unsigned int inf_stop_at_err_count = 0;+ /* Nonzero means check symlink mtimes.  */  int check_symlink_flag = 0;@@ -394,6 +398,8 @@   -S, --no-keep-going, --stop\n\                               Turns off -k.\n"),     N_("\+      --stop-at-err-count[=N] Stop when N errors are reached. Default 0 (infinite).\n"),+    N_("\   -t, --touch                 Touch targets instead of remaking them.\n"),     N_("\   --trace                     Print tracing information.\n"),@@ -471,6 +477,9 @@       "warn-undefined-variables" },     { CHAR_MAX+6, strlist, &eval_strings, 1, 0, 0, 0, 0, "eval" },     { CHAR_MAX+7, string, &sync_mutex, 1, 1, 0, 0, 0, "sync-mutex" },+    { CHAR_MAX+8, positive_int, (char *) &max_stop_at_err_count, 1, 1, 0,+      (char *) &inf_stop_at_err_count,+      (char *) &max_stop_at_err_count, "stop-at-err-count" },     { 0, 0, 0, 0, 0, 0, 0, 0, 0 }   }; diff -u /home/foo/make_git/make/makeint.h make/make-4.2.1/makeint.h--- /home/foo/make_git/make/makeint.h 2018-10-04 16:30:01.169374000 -0400+++ make/make-4.2.1/makeint.h 2018-10-04 14:15:07.587723000 -0400@@ -639,6 +639,9 @@ extern const floc *reading_file; extern const floc **expanding_var; +extern int stop_at_err_count;+extern unsigned int max_stop_at_err_count;+ extern unsigned short stopchar_map[];  extern int just_print_flag, silent_flag, ignore_errors_flag, keep_going_flag;


Regards Richa Bindra
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Re: proposal: stop_at_err_count patch

Martin d'Anjou-5
Hello,

Richa: Somehow the email formatting screw up the patch, can you try to send
it as an attachment instead?

I had a similar patch on GNU Make 3.81, thanks for making it for 4.2.1.
This saves thousands of hours of CPU on the CI host, hours that can go to
other tasks. I also found my old bug report (
https://savannah.gnu.org/bugs/?41781).

What I'd like to know from Paul and other owners/contributors: what would
be required to make such a feature part of GNU Make?

Best,
Martin d'Anjou



On Thu, Oct 4, 2018 at 5:04 PM Richa Bindra <[hidden email]> wrote:

> Hi,
>
> Here is a small patch that adds a new option to make that causes make to
> abort after N failures. An extra argument --stop-at-err-count[=N] is
> provided which forces GNU Make to return immediately when the number of
> errors is reached.
> The current problem with this patch is that when GNU make exits, the
> children processes left are not killed by GNU make when it exits. I want
> all children to be killed before GNU Make returns. I am not well versed
> with GNU Make code, so any advice or feedback would be appreciated!
>
> diff -u /home/foo/make_git/make/job.c make/make-4.2.1/job.c---
> /home/foo/make_git/make/job.c 2018-10-04 16:30:01.108348000 -0400+++
> make/make-4.2.1/job.c 2018-10-04 16:28:12.119052000 -0400@@ -478,9
> +478,19 @@   const char *nm;   size_t l; +  if (max_stop_at_err_count <=
> stop_at_err_count)+  {+    error (NILF, 0, _("*** [%p] Error (aborted after
> %d failures) %d"),+        child, stop_at_err_count, exit_code);+    die
> (2);+  }+   if (ignored && silent_flag)     return; +/* Only increase
> STOP_AT_ERR_COUNT if the job _really_ failed.  */+  stop_at_err_count++;+
>  if (exit_sig && coredump)     dump = _(" (core dumped)"); @@ -595,8
> +605,14 @@           /* We might block for a while, so let the user know
> why.              Only print this message once no matter how many jobs are
> left.  */           fflush (stdout);-          if (!printed)-            O
> (error, NILF, _("*** Waiting for unfinished jobs...."));+          if
> (!printed) {+             O (error, NILF, _("*** Waiting for unfinished
> jobs...."));+             if (max_stop_at_err_count <= stop_at_err_count)+
>              {+                 error (NILF,0, _("*** Aborting after %d
> failures"),stop_at_err_count);+                 die (2);+               }+
>         }           printed = 1;         } diff -u
> /home/foo/make_git/make/main.c make/make-4.2.1/main.c---
> /home/foo/make_git/make/main.c 2018-10-04 16:30:01.120391000 -0400+++
> make/make-4.2.1/main.c 2018-10-04 16:29:19.185057000 -0400@@ -216,9
> +216,13 @@  /* Nonzero means keep going even if remaking some file fails
> (-k).  */ -int keep_going_flag;+int keep_going_flag = 0; int
> default_keep_going_flag = 0; +int stop_at_err_count = 0;+unsigned int
> max_stop_at_err_count = -1;+static unsigned int inf_stop_at_err_count =
> 0;+ /* Nonzero means check symlink mtimes.  */  int check_symlink_flag = 0;@@
> -394,6 +398,8 @@   -S, --no-keep-going, --stop\n\
>      Turns off -k.\n"),     N_("\+      --stop-at-err-count[=N] Stop when N
> errors are reached. Default 0 (infinite).\n"),+    N_("\   -t, --touch
>            Touch targets instead of remaking them.\n"),     N_("\
>  --trace                     Print tracing information.\n"),@@ -471,6
> +477,9 @@       "warn-undefined-variables" },     { CHAR_MAX+6, strlist,
> &eval_strings, 1, 0, 0, 0, 0, "eval" },     { CHAR_MAX+7, string,
> &sync_mutex, 1, 1, 0, 0, 0, "sync-mutex" },+    { CHAR_MAX+8, positive_int,
> (char *) &max_stop_at_err_count, 1, 1, 0,+      (char *)
> &inf_stop_at_err_count,+      (char *) &max_stop_at_err_count,
> "stop-at-err-count" },     { 0, 0, 0, 0, 0, 0, 0, 0, 0 }   }; diff -u
> /home/foo/make_git/make/makeint.h make/make-4.2.1/makeint.h---
> /home/foo/make_git/make/makeint.h 2018-10-04 16:30:01.169374000 -0400+++
> make/make-4.2.1/makeint.h 2018-10-04 14:15:07.587723000 -0400@@ -639,6
> +639,9 @@ extern const floc *reading_file; extern const floc
> **expanding_var; +extern int stop_at_err_count;+extern unsigned int
> max_stop_at_err_count;+ extern unsigned short stopchar_map[];  extern int
> just_print_flag, silent_flag, ignore_errors_flag, keep_going_flag;
>
>
> Regards Richa Bindra
> _______________________________________________
> Help-make mailing list
> [hidden email]
> https://lists.gnu.org/mailman/listinfo/help-make
>
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make
Reply | Threaded
Open this post in threaded view
|

Fw: proposal: stop_at_err_count patch

Richa Bindra
Hi
 Thanks Martin for pointing the formatting issue.

The patch I posted earlier is poorly formatted, so I am trying an attachment instead.

The current problem with this patch is that when GNU make exits, the children processes left are not killed by GNU make when it exits. I want all children to be killed before GNU Make returns. I am not well versed with GNU Make code, so any advice or feedback would be appreciated!


Regards
Richa Bindra

   ----- Forwarded message ----- From: Martin d'Anjou <[hidden email]>To: Richa Bindra <[hidden email]>Cc: "[hidden email]" <[hidden email]>Sent: Monday, 8 October, 2018, 9:41:26 AM GMT-4Subject: Re: proposal: stop_at_err_count patch
 Hello,
Richa: Somehow the email formatting screw up the patch, can you try to send it as an attachment instead?

I had a similar patch on GNU Make 3.81, thanks for making it for 4.2.1. This saves thousands of hours of CPU on the CI host, hours that can go to other tasks. I also found my old bug report (https://savannah.gnu.org/bugs/?41781).
What I'd like to know from Paul and other owners/contributors: what would be required to make such a feature part of GNU Make?
Best,Martin d'Anjou


On Thu, Oct 4, 2018 at 5:04 PM Richa Bindra <[hidden email]> wrote:

Hi,

Here is a small patch that adds a new option to make that causes make to abort after N failures. An extra argument --stop-at-err-count[=N] is provided which forces GNU Make to return immediately when the number of errors is reached. 
The current problem with this patch is that when GNU make exits, the children processes left are not killed by GNU make when it exits. I want all children to be killed before GNU Make returns. I am not well versed with GNU Make code, so any advice or feedback would be appreciated!

diff -u /home/foo/make_git/make/job.c make/make-4.2.1/job.c--- /home/foo/make_git/make/job.c 2018-10-04 16:30:01.108348000 -0400+++ make/make-4.2.1/job.c 2018-10-04 16:28:12.119052000 -0400@@ -478,9 +478,19 @@   const char *nm;   size_t l; +  if (max_stop_at_err_count <= stop_at_err_count)+  {+    error (NILF, 0, _("*** [%p] Error (aborted after %d failures) %d"),+        child, stop_at_err_count, exit_code);+    die (2);+  }+   if (ignored && silent_flag)     return; +/* Only increase STOP_AT_ERR_COUNT if the job _really_ failed.  */+  stop_at_err_count++;+   if (exit_sig && coredump)     dump = _(" (core dumped)"); @@ -595,8 +605,14 @@           /* We might block for a while, so let the user know why.              Only print this message once no matter how many jobs are left.  */           fflush (stdout);-          if (!printed)-            O (error, NILF, _("*** Waiting for unfinished jobs...."));+          if (!printed) {+             O (error, NILF, _("*** Waiting for unfinished jobs...."));+             if (max_stop_at_err_count <= stop_at_err_count)+               {+                 error (NILF,0, _("*** Aborting after %d failures"),stop_at_err_count);+                 die (2);+               }+          }           printed = 1;         } diff -u /home/foo/make_git/make/main.c make/make-4.2.1/main.c--- /home/foo/make_git/make/main.c 2018-10-04 16:30:01.120391000 -0400+++ make/make-4.2.1/main.c 2018-10-04 16:29:19.185057000 -0400@@ -216,9 +216,13 @@  /* Nonzero means keep going even if remaking some file fails (-k).  */ -int keep_going_flag;+int keep_going_flag = 0; int default_keep_going_flag = 0; +int stop_at_err_count = 0;+unsigned int max_stop_at_err_count = -1;+static unsigned int inf_stop_at_err_count = 0;+ /* Nonzero means check symlink mtimes.  */  int check_symlink_flag = 0;@@ -394,6 +398,8 @@   -S, --no-keep-going, --stop\n\                               Turns off -k.\n"),     N_("\+      --stop-at-err-count[=N] Stop when N errors are reached. Default 0 (infinite).\n"),+    N_("\   -t, --touch                 Touch targets instead of remaking them.\n"),     N_("\   --trace                     Print tracing information.\n"),@@ -471,6 +477,9 @@       "warn-undefined-variables" },     { CHAR_MAX+6, strlist, &eval_strings, 1, 0, 0, 0, 0, "eval" },     { CHAR_MAX+7, string, &sync_mutex, 1, 1, 0, 0, 0, "sync-mutex" },+    { CHAR_MAX+8, positive_int, (char *) &max_stop_at_err_count, 1, 1, 0,+      (char *) &inf_stop_at_err_count,+      (char *) &max_stop_at_err_count, "stop-at-err-count" },     { 0, 0, 0, 0, 0, 0, 0, 0, 0 }   }; diff -u /home/foo/make_git/make/makeint.h make/make-4.2.1/makeint.h--- /home/foo/make_git/make/makeint.h 2018-10-04 16:30:01.169374000 -0400+++ make/make-4.2.1/makeint.h 2018-10-04 14:15:07.587723000 -0400@@ -639,6 +639,9 @@ extern const floc *reading_file; extern const floc **expanding_var; +extern int stop_at_err_count;+extern unsigned int max_stop_at_err_count;+ extern unsigned short stopchar_map[];  extern int just_print_flag, silent_flag, ignore_errors_flag, keep_going_flag;


Regards Richa Bindra
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make

 
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make

make.patch (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

proposal: stop_at_err_count patch

Richa Bindra
Hi
Just wondering if anyone is interested in the proposed patch?

Regards
Richa Bindra

   ----- Forwarded message ----- From: Richa Bindra <[hidden email]>To: [hidden email] <[hidden email]>Sent: Tuesday, 9 October, 2018, 9:32:30 AM GMT-4Subject: Fw: proposal: stop_at_err_count patch
 Hi
 Thanks Martin for pointing the formatting issue.

The patch I posted earlier is poorly formatted, so I am trying an attachment instead.

The current problem with this patch is that when GNU make exits, the children processes left are not killed by GNU make when it exits. I want all children to be killed before GNU Make returns. I am not well versed with GNU Make code, so any advice or feedback would be appreciated!


Regards
Richa Bindra

   ----- Forwarded message ----- From: Martin d'Anjou <[hidden email]>To: Richa Bindra <[hidden email]>Cc: "[hidden email]" <[hidden email]>Sent: Monday, 8 October, 2018, 9:41:26 AM GMT-4Subject: Re: proposal: stop_at_err_count patch
 Hello,
Richa: Somehow the email formatting screw up the patch, can you try to send it as an attachment instead?

I had a similar patch on GNU Make 3.81, thanks for making it for 4.2.1. This saves thousands of hours of CPU on the CI host, hours that can go to other tasks. I also found my old bug report (https://savannah.gnu.org/bugs/?41781).
What I'd like to know from Paul and other owners/contributors: what would be required to make such a feature part of GNU Make?
Best,Martin d'Anjou


On Thu, Oct 4, 2018 at 5:04 PM Richa Bindra <[hidden email]> wrote:

Hi,

Here is a small patch that adds a new option to make that causes make to abort after N failures. An extra argument --stop-at-err-count[=N] is provided which forces GNU Make to return immediately when the number of errors is reached. 
The current problem with this patch is that when GNU make exits, the children processes left are not killed by GNU make when it exits. I want all children to be killed before GNU Make returns. I am not well versed with GNU Make code, so any advice or feedback would be appreciated!

diff -u /home/foo/make_git/make/job.c make/make-4.2.1/job.c--- /home/foo/make_git/make/job.c 2018-10-04 16:30:01.108348000 -0400+++ make/make-4.2.1/job.c 2018-10-04 16:28:12.119052000 -0400@@ -478,9 +478,19 @@   const char *nm;   size_t l; +  if (max_stop_at_err_count <= stop_at_err_count)+  {+    error (NILF, 0, _("*** [%p] Error (aborted after %d failures) %d"),+        child, stop_at_err_count, exit_code);+    die (2);+  }+   if (ignored && silent_flag)     return; +/* Only increase STOP_AT_ERR_COUNT if the job _really_ failed.  */+  stop_at_err_count++;+   if (exit_sig && coredump)     dump = _(" (core dumped)"); @@ -595,8 +605,14 @@           /* We might block for a while, so let the user know why.              Only print this message once no matter how many jobs are left.  */           fflush (stdout);-          if (!printed)-            O (error, NILF, _("*** Waiting for unfinished jobs...."));+          if (!printed) {+             O (error, NILF, _("*** Waiting for unfinished jobs...."));+             if (max_stop_at_err_count <= stop_at_err_count)+               {+                 error (NILF,0, _("*** Aborting after %d failures"),stop_at_err_count);+                 die (2);+               }+          }           printed = 1;         } diff -u /home/foo/make_git/make/main.c make/make-4.2.1/main.c--- /home/foo/make_git/make/main.c 2018-10-04 16:30:01.120391000 -0400+++ make/make-4.2.1/main.c 2018-10-04 16:29:19.185057000 -0400@@ -216,9 +216,13 @@  /* Nonzero means keep going even if remaking some file fails (-k).  */ -int keep_going_flag;+int keep_going_flag = 0; int default_keep_going_flag = 0; +int stop_at_err_count = 0;+unsigned int max_stop_at_err_count = -1;+static unsigned int inf_stop_at_err_count = 0;+ /* Nonzero means check symlink mtimes.  */  int check_symlink_flag = 0;@@ -394,6 +398,8 @@   -S, --no-keep-going, --stop\n\                               Turns off -k.\n"),     N_("\+      --stop-at-err-count[=N] Stop when N errors are reached. Default 0 (infinite).\n"),+    N_("\   -t, --touch                 Touch targets instead of remaking them.\n"),     N_("\   --trace                     Print tracing information.\n"),@@ -471,6 +477,9 @@       "warn-undefined-variables" },     { CHAR_MAX+6, strlist, &eval_strings, 1, 0, 0, 0, 0, "eval" },     { CHAR_MAX+7, string, &sync_mutex, 1, 1, 0, 0, 0, "sync-mutex" },+    { CHAR_MAX+8, positive_int, (char *) &max_stop_at_err_count, 1, 1, 0,+      (char *) &inf_stop_at_err_count,+      (char *) &max_stop_at_err_count, "stop-at-err-count" },     { 0, 0, 0, 0, 0, 0, 0, 0, 0 }   }; diff -u /home/foo/make_git/make/makeint.h make/make-4.2.1/makeint.h--- /home/foo/make_git/make/makeint.h 2018-10-04 16:30:01.169374000 -0400+++ make/make-4.2.1/makeint.h 2018-10-04 14:15:07.587723000 -0400@@ -639,6 +639,9 @@ extern const floc *reading_file; extern const floc **expanding_var; +extern int stop_at_err_count;+extern unsigned int max_stop_at_err_count;+ extern unsigned short stopchar_map[];  extern int just_print_flag, silent_flag, ignore_errors_flag, keep_going_flag;


Regards Richa Bindra
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make

   
_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make

make.patch (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: proposal: stop_at_err_count patch

Paul Smith-20
On Mon, 2018-10-22 at 13:19 +0000, Richa Bindra wrote:
> Just wondering if anyone is interested in the proposed patch?

I have no problem with having a mode or method of causing the entire
build to fail quickly whenever the first job fails.  However, whatever
method that is has to be complete and work in all situations, including
recursive make, and needs a way to stop all child processes before
exiting so they don't continue to run after make exits.

And any new feature like this needs to have tests available.

I'm not suggesting that any one person needs to do all this, I'm just
explaining why I've not applied the proposed changes, as they are not
complete.


For this particular change, is there a good justification for allowing
some number of errors before failing?  In what situation would you want
to keep going after three errors, but fail if you hit a fourth (for
example)?  I'm not sure I see the need for that feature.

In my opinion these are the two important enhancements:

One: if a failure happens somewhere in the build stop processing
cleanly as soon as possible... by that I mean, allow running jobs to
complete but don't start any new jobs.  This is how the build works
today BUT it is not distributed across recursive invocations.  That
should be fixed and this should be the default behavior (if -k is not
given of course).

Two: if a failure happens somewhere in the build stop processing
immediately... by this I mean, kill all currently running child jobs
and exit right away.  There can be some extra option to enable this
behavior.  We will probably want to consider deleting the target files
for any jobs we kill... that will need consideration.  Not all recipes
or tools will be careful to do things like write to temporary files and
atomically rename once the file is complete.


_______________________________________________
Help-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/help-make