patch locale settings

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

patch locale settings

Gnu - Make - Bugs mailing list
Multiple tests fail when locale related env variables (such as LANG or
LC_MESSAGES) are set to some regional setting.

regards, Dmitry

diff --git a/tests/run_make_tests b/tests/run_make_tests
index b68b784..7b3719e 100755
--- a/tests/run_make_tests
+++ b/tests/run_make_tests
@@ -1,2 +1,3 @@
 #!/bin/sh
+export LC_ALL=C
 exec perl $0.pl ${1+"$@"}

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Paul Smith-20
On Fri, 2019-09-13 at 21:43 -0400, Dmitry Goncharov via Bug reports and
discussion for GNU make wrote:
> Multiple tests fail when locale related env variables (such as LANG or
> LC_MESSAGES) are set to some regional setting.

This should not be necessary because we already do this, in
test_driver.pl:

  # Make sure our compares are not foiled by locale differences

  $makeENV{LC_ALL} = 'C';

and, I've checked with a makefile invoked by the test suite and this
setting is in effect, even if the shell invoking run_make_tests.pl has
a different setting for LC_ALL.

Furthermore, we clear out the environment so that only a set number of
variables are actually reserved, and LANG is not one of them (nor is
LC_MESSAGES), so setting LANG in the run_make_tests shell script should
not have any effect.

I need to understand your failure situation better before applying this
change.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Gnu - Make - Bugs mailing list
On Sat, Sep 14, 2019 at 10:22 AM Paul Smith <[hidden email]> wrote:
> and, I've checked with a makefile invoked by the test suite and this
> setting is in effect, even if the shell invoking run_make_tests.pl has
> a different setting for LC_ALL.

This should be sufficient.

> Furthermore, we clear out the environment so that only a set number of
> variables are actually reserved, and LANG is not one of them (nor is
> LC_MESSAGES), so setting LANG in the run_make_tests shell script should
> not have any effect.

LANG and LC_MESSAGES both affect tests on certain hosts.

> I need to understand your failure situation better before applying this
> change.

Paul, i assume the system on which you test can print messages in various
locales. I have a host which cannot do anything other than the default locale.
Tests pass on that host regardless of env variables.

Here is an example from a host which honors env variables
$ export LC_MESSAGES=fr_CA.UTF-8; echo 'all: ; ./asdf' | ../l64/make -f-
./asdf
make: ./asdf: Aucun fichier ou dossier de ce type
make: *** [/tmp/GmvTj9K1;1: all] Error 127

What does
export LC_MESSAGES=fr_CA.UTF-8; echo 'all: ; ./asdf' | ../l64/make -f-
print on your host?

Below are a couple of examples of failures on aix and linux.
These tests pass when LANG and LC_MESSAGES are unset or set to C.

$ LANG=en_US ./run_make_tests -make_path ../i32/make features/errors;
cat work/features/errors.diff.3
------------------------------------------------------------------------------
            Running tests for GNU make on AIX 1 00F85A9D4C00
                               GNU Make 4.2.90
------------------------------------------------------------------------------

Clearing work...
Making work dirs...

features/errors ......................................... FAILED (4/7 passed)

3 Tests in 1 Category Failed (See .diff* files in work dir for details) :-(

*** work/features/errors.base.3 Sat Sep 14 15:16:57 2019
--- work/features/errors.log.3  Sat Sep 14 15:16:57 2019
***************
*** 1,3 ****
  ./foobarbazbozblat xx yy
! make: ./foobarbazbozblat: A file or directory in the path name does not exist.
  make: [work/features/errors.mk.2;2: one] Error 127 (ignored)
--- 1,3 ----
  ./foobarbazbozblat xx yy
! make: ./foobarbazbozblat: No such file or directory
  make: [work/features/errors.mk.2;2: one] Error 127 (ignored)
$



$ LC_MESSAGES=ru_RU ./run_make_tests -make_path ../l32/make
features/errors; cat work/features/errors.diff.3
------------------------------------------------------------------------------
Running tests for GNU make on Linux 2.6.32-642.6.2.el6.x86_64 x86_64
                               GNU Make 4.2.90
------------------------------------------------------------------------------

Clearing work...
Making work dirs...

features/errors ......................................... FAILED (4/7 passed)

3 Tests in 1 Category Failed (See .diff* files in work dir for details) :-(

*** work/features/errors.base.3 Sat Sep 14 15:19:22 2019
--- work/features/errors.log.3  Sat Sep 14 15:19:22 2019
***************
*** 1,3 ****
  ./foobarbazbozblat xx yy
! make: ./foobarbazbozblat: ??? ?????? ????? ??? ????????
  make: [work/features/errors.mk.2;2: one] Error 127 (ignored)
--- 1,3 ----
  ./foobarbazbozblat xx yy
! make: ./foobarbazbozblat: No such file or directory
  make: [work/features/errors.mk.2;2: one] Error 127 (ignored)


regards, Dmitry

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Paul Smith-20
On Sat, 2019-09-14 at 16:19 -0400, Dmitry Goncharov wrote:
> On Sat, Sep 14, 2019 at 10:22 AM Paul Smith <[hidden email]> wrote:
> > and, I've checked with a makefile invoked by the test suite and
> > this setting is in effect, even if the shell invoking
> > run_make_tests.pl has a different setting for LC_ALL.
>
> This should be sufficient.

It should be, yes, but based on your results below it is not.  This is
why I'm confused.

> > Furthermore, we clear out the environment so that only a set number
> > of variables are actually reserved, and LANG is not one of them
> > (nor is LC_MESSAGES), so setting LANG in the run_make_tests shell
> > script should not have any effect.
>
> LANG and LC_MESSAGES both affect tests on certain hosts.

Well, first the LC_ALL value is supposed to supersede ALL other LC_*
values and also LANG.  So simply setting LC_ALL (as we do) should be
sufficient to make the values of both LANG and LC_MESSAGES irrelevant.

Second, the test framework throws out everything in the user's
environment other than a few specific values when it runs make from
within a test.  So it shouldn't matter that you have set either LANG or
LC_MESSAGES, or anything else, before you started the tests.

> > I need to understand your failure situation better before applying
> > this change.
>
> Here is an example from a host which honors env variables
> $ export LC_MESSAGES=fr_CA.UTF-8; echo 'all: ; ./asdf' | ../l64/make
> -f-
> ./asdf
> make: ./asdf: Aucun fichier ou dossier de ce type
> make: *** [/tmp/GmvTj9K1;1: all] Error 127

Sure, but none of those instances of make are invoked from within our
test suite.

> What does
> export LC_MESSAGES=fr_CA.UTF-8; echo 'all: ; ./asdf' | ../l64/make -f-
> print on your host?

I don't have French installed but I do have German:

  $ echo 'all:;./asdf' | LC_MESSAGES=de_DE.UTF-8 ./make -f-
  ./asdf
  make: ./asdf: Datei oder Verzeichnis nicht gefunden
  make: *** [/tmp/GmIlXXqu;1: all] Fehler 127

> Below are a couple of examples of failures on aix and linux.
> These tests pass when LANG and LC_MESSAGES are unset or set to C.

This is what I can't understand.  Your setting of LANG or LC_MESSAGES
cannot make any difference to the test suite because the test suite
does not pass those values to the make that it invokes.

> $ LC_MESSAGES=ru_RU ./run_make_tests -make_path ../l32/make

I'm surprised that even works; on my system if I use a shorthand like
"de_DE" instead of the full "de_DE.UTF-8", then it isn't recognized and
I get the default.

> features/errors; cat work/features/errors.diff.3
>
> 3 Tests in 1 Category Failed (See .diff* files in work dir for
> details) :-(
>
> *** work/features/errors.base.3 Sat Sep 14 15:19:22 2019
> --- work/features/errors.log.3  Sat Sep 14 15:19:22 2019
> ***************
> *** 1,3 ****
>   ./foobarbazbozblat xx yy
> ! make: ./foobarbazbozblat: ??? ?????? ????? ??? ????????
>   make: [work/features/errors.mk.2;2: one] Error 127 (ignored)
> --- 1,3 ----
>   ./foobarbazbozblat xx yy
> ! make: ./foobarbazbozblat: No such file or directory
>   make: [work/features/errors.mk.2;2: one] Error 127 (ignored)
I can't understand this.  If I run my tests like this they pass just as
expected:

  $ LC_MESSAGES=de_DE.UTF-8 ./run_make_tests -make ../make features/errors
    ...
  9 Tests in 1 Category Complete ... No Failures :-)

Can you apply the attached patch, then re-run the test features/errors
and send me the diff output?

FYI, this is what I get:

$ cat work/features/errors.diff.3
*** work/features/errors.base.3 Sat Sep 14 18:19:42 2019
--- work/features/errors.log.3  Sat Sep 14 18:19:42 2019
***************
*** 1,3 ****
--- 1,13 ----
+ env
+ LC_ALL=C
+ MAKEFLAGS=
+ MFLAGS=
+ SHELL=/bin/bash
+ PATH=/...
+ LOGNAME=psmith
+ USER=psmith
+ HOME=/home/psmith
+ MAKELEVEL=1
  ./foobarbazbozblat xx yy
  make: ./foobarbazbozblat: No such file or directory
  make: [work/features/errors.mk.2;2: one] Error 127 (ignored)

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make

errors.diff (856 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Paul Smith-20
On Sat, 2019-09-14 at 18:27 -0400, Paul Smith wrote:
Can you apply the attached patch

Oops ignore/delete the diff of m4/.gitignore :(


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Gnu - Make - Bugs mailing list
In reply to this post by Paul Smith-20
On Sat, Sep 14, 2019 at 6:27 PM Paul Smith <[hidden email]> wrote:
> Your setting of LANG or LC_MESSAGES
> cannot make any difference to the test suite because the test suite
> does not pass those values to the make that it invokes.

Paul, that's the output from the perl program (called base) that is
affected by the env variable. The output from gmake is all right.
$ERR_no_such_file is initialized in test_driver.pl on line 75. This
initialization takes place still in the env inherited by the perl
program, before resetENV runs.

i applied this patch
diff --git a/tests/test_driver.pl b/tests/test_driver.pl
index 4af84da..0260357 100644
--- a/tests/test_driver.pl
+++ b/tests/test_driver.pl
@@ -75,2 +75,3 @@ if (open(my $F, '<', 'file.none')) {
     $ERR_no_such_file = "$!";
+    print "lc_messages=$ENV{LC_MESSAGES}, set ERR_no_such_file to
$ERR_no_such_file\n";
 }
@@ -172,2 +173,3 @@ sub resetENV
   }
+  print "reset env, lc_messages=$ENV{LC_MESSAGES}\n";
 }

And here are the first 3 lines of output
$ LC_MESSAGES=ru_RU ./run_make_tests -make_path ../l64/make features/errors
lc_messages=ru_RU, set ERR_no_such_file to ??? ?????? ????? ??? ????????
reset env, lc_messages=
------------------------------------------------------------------------------

The only question is how the output from the perl program (called
base) is not affected by env variable on your host.
regards, Dmitry

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Paul Smith-20
On Sat, 2019-09-14 at 22:31 -0400, Dmitry Goncharov wrote:
> On Sat, Sep 14, 2019 at 6:27 PM Paul Smith <[hidden email]> wrote:
> > Your setting of LANG or LC_MESSAGES
> > cannot make any difference to the test suite because the test suite
> > does not pass those values to the make that it invokes.
>
> Paul, that's the output from the perl program (called base) that is
> affected by the env variable. The output from gmake is all right.

I see.  However, that should not be the case either (and in fact, as I
showed in my previous email, it doesn't fail for me when I explicitly
set LC_MESSAGES to a different language on my system).

That's because before we try to detect these messages we explicitly set
the locale in Perl; see test_driver.pl:

  my $loc = undef;
  if ($has_POSIX) {
      $loc = POSIX::setlocale(POSIX::LC_MESSAGES);
      POSIX::setlocale(POSIX::LC_MESSAGES, 'C');
  }
   ...get messages...
  $loc and POSIX::setlocale(POSIX::LC_MESSAGES, $loc);

So, the question is why does your Perl still provide language-specific
messages?

Hm.  It might be that we need to run "use locale;" as well.  It doesn't
seem necessary for me but maybe...?

FWIW, I don't want to change the shell script and call that done
because not everyone invokes the unit tests via the shell script: they
may invoke it via "perl run_make_tests.pl ..." directly.


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Gnu - Make - Bugs mailing list
On Sun, Sep 15, 2019 at 08:31:39AM -0400, Paul Smith wrote:

> That's because before we try to detect these messages we explicitly set
> the locale in Perl; see test_driver.pl:
>
>   my $loc = undef;
>   if ($has_POSIX) {
>       $loc = POSIX::setlocale(POSIX::LC_MESSAGES);
>       POSIX::setlocale(POSIX::LC_MESSAGES, 'C');
>   }
>    ...get messages...
>   $loc and POSIX::setlocale(POSIX::LC_MESSAGES, $loc);
>
> So, the question is why does your Perl still provide language-specific
> messages?
has_POSIX is uninitialized at the time test_driver.pl reads it, because
(atleast in my testing) the code which initializes has_POSIX hasn't run yet.

...

> FWIW, I don't want to change the shell script and call that done
> because not everyone invokes the unit tests via the shell script: they
> may invoke it via "perl run_make_tests.pl ..." directly.
Let me come up with a fix for 'perl run_make_tests.pl'.

regards, Dmitry


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Paul Smith-20
On Sun, 2019-09-15 at 09:44 -0400, Dmitry Goncharov wrote:
> > So, the question is why does your Perl still provide language-specific
> > messages?
> has_POSIX is uninitialized at the time test_driver.pl reads it, because
> (atleast in my testing) the code which initializes has_POSIX hasn't run yet.

I see.  I wonder why it works for me.

> > FWIW, I don't want to change the shell script and call that done
> > because not everyone invokes the unit tests via the shell script: they
> > may invoke it via "perl run_make_tests.pl ..." directly.
> Let me come up with a fix for 'perl run_make_tests.pl'.

Never mind, I can fix it.

Thanks for helping me debug!


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Gnu - Make - Bugs mailing list
On Sun, Sep 15, 2019 at 10:14:07AM -0400, Paul Smith wrote:

> I see.  I wonder why it works for me.
i was thinking your system couldn't do anything other than the default, but your
german test demonstrates it can.
>
> > Let me come up with a fix for 'perl run_make_tests.pl'.

if you set
use strict;
use warnings;
perl will tell you that it does not know what POSIX::LC_MESSAGES is.

This patch initializes has_posix right here in test_driver.pl and pulls in locale_h.
This patch sets LC_ALL because we saw that LANG has an affect.
This patch won't help those who don't have POSIX.pm (windows?).
I suggest we still set LC_ALL=C in the shell program for systems w/o POSIX.pm.


diff --git a/tests/test_driver.pl b/tests/test_driver.pl
index 4af84da..fb3f0a5 100644
--- a/tests/test_driver.pl
+++ b/tests/test_driver.pl
@@ -60,9 +60,11 @@ $perl_name =~ tr,\\,/,;
 # We want them from the C locale regardless of our current locale.
 
 my $loc = undef;
-if ($has_POSIX) {
-    $loc = POSIX::setlocale(POSIX::LC_MESSAGES);
-    POSIX::setlocale(POSIX::LC_MESSAGES, 'C');
+my $has_posix = eval { require "POSIX.pm" };
+if ($has_posix) {
+    use POSIX 'locale_h';
+    $loc = POSIX::setlocale(POSIX::LC_ALL);
+    POSIX::setlocale(POSIX::LC_ALL, 'C');
 }
 
 $ERR_no_such_file = undef;
@@ -96,7 +98,7 @@ if (open(my $F, '<', 'file.out')) {
 
 unlink('file.out') or die "Failed to delete file.out: $!\n";
 
-$loc and POSIX::setlocale(POSIX::LC_MESSAGES, $loc);
+$loc and POSIX::setlocale(POSIX::LC_ALL, $loc);
 
 # %makeENV is the cleaned-out environment.
 %makeENV = ();



> Thanks for helping me debug!
Thank you, Paul, for your work on gmake. i'll be glad to help.

regards, Dmitry
>

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Paul Smith-20
On Sun, 2019-09-15 at 13:01 -0400, Dmitry Goncharov wrote:
> perl will tell you that it does not know what POSIX::LC_MESSAGES is.
>
> This patch initializes has_posix right here in test_driver.pl and pulls in locale_h.
> This patch sets LC_ALL because we saw that LANG has an affect.
> This patch won't help those who don't have POSIX.pm (windows?).

I went a slightly different way: test_driver.pl is (theoretically)
supposed to be a base driver that doesn't know anything about make
itself, so I moved all the error string detection into run_make_tests.

After importing properly LC_MESSAGES worked on UNIX, but I switched to
using LC_ALL because Windows does have POSIX locale but only LC_ALL.

I hope this will work now.  I still don't know why I couldn't reproduce
the failure myself.

Thanks!


_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make
Reply | Threaded
Open this post in threaded view
|

Re: patch locale settings

Andreas Schwab
In reply to this post by Paul Smith-20
On Sep 14 2019, Paul Smith <[hidden email]> wrote:

> On Sat, 2019-09-14 at 16:19 -0400, Dmitry Goncharov wrote:
>> $ LC_MESSAGES=ru_RU ./run_make_tests -make_path ../l32/make
>
> I'm surprised that even works; on my system if I use a shorthand like
> "de_DE" instead of the full "de_DE.UTF-8", then it isn't recognized and
> I get the default.

ru_RU (or de_DE) is just another locale that you can install (in glibc,
it uses ISO-8859-5 as the codeset, or ISO-8859-1 for the de_DE locale).

Andreas.

--
Andreas Schwab, SUSE Labs, [hidden email]
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

_______________________________________________
Bug-make mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/bug-make