Bug 16119 - "remote: FATAL: W * lebarhon DENIED by fallthru" error whenever lebarhon tries to push to git
Summary: "remote: FATAL: W * lebarhon DENIED by fallthru" error whenever lebarhon trie...
Status: RESOLVED FIXED
Alias: None
Product: Infrastructure
Classification: Unclassified
Component: Others (show other bugs)
Version: unspecified
Hardware: All Linux
Priority: Normal normal
Target Milestone: ---
Assignee: Colin Guthrie
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-06-15 15:47 CEST by Marja Van Waes
Modified: 2015-07-15 13:02 CEST (History)
3 users (show)

See Also:
Source RPM:
CVE:
Status comment:


Attachments

Description Marja Van Waes 2015-06-15 15:47:08 CEST
I should have asked lebarhon to file a new bug report, instead of using the bug that was used to give him commit rights, because, as far as we know it has worked with svn.

In spite of numerous clean checkouts, whenever he tries to push a change to software/i18n/tools/ or a testfile to software/drakx-installer-help, lebarhon gets something like:

[sam4@localhost fr (master)]$ git push      
X11 forwarding request failed on channel 0
Counting objects: 4, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 331 bytes | 0 bytes/s, done.
Total 4 (delta 2), reused 1 (delta 0)
remote: FATAL: W refs/heads/master software/drakx-installer-help lebarhon DENIED by fallthru
remote: error: hook declined to update refs/heads/master
To ssh://git@git.mageia.org/software/drakx-installer-help
 ! [remote rejected] master -> master (hook declined)
error: failed to push some refs to 'ssh://git@git.mageia.org/software/drakx-installer-help'
[sam4@localhost fr (master)]$

@ lebarhon

Do you mind attaching the output of

   ssh -v lebarhon@svn.mageia.org
Comment 1 André DESMOTTES 2015-06-15 22:18:53 CEST
Here it is:

[samageia@localhost ~]$  ssh -v lebarhon@svn.mageia.org
OpenSSH_6.2p2, OpenSSL 1.0.1m 19 Mar 2015
debug1: Reading configuration data /home/samageia/.ssh/config
debug1: /home/samageia/.ssh/config line 1: Applying options for *.mageia.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 49: Applying options for *
debug1: Connecting to svn.mageia.org [212.85.158.147] port 22.
debug1: Connection established.
debug1: identity file /home/samageia/.ssh/identity type -1
debug1: identity file /home/samageia/.ssh/identity-cert type -1
debug1: identity file /home/samageia/.ssh/id_rsa type 1
debug1: identity file /home/samageia/.ssh/id_rsa-cert type -1
debug1: identity file /home/samageia/.ssh/id_dsa type -1
debug1: identity file /home/samageia/.ssh/id_dsa-cert type -1
debug1: identity file /home/samageia/.ssh/id_ecdsa type -1
debug1: identity file /home/samageia/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.8
debug1: match: OpenSSH_5.8 pat OpenSSH_5*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: RSA ad:34:4d:3f:37:73:91:51:9b:af:80:ce:e9:d6:2a:a4
debug1: Host 'svn.mageia.org' is known and matches the RSA host key.
debug1: Found key in /home/samageia/.ssh/known_hosts:7
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/samageia/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to svn.mageia.org ([212.85.158.147]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Requesting X11 forwarding with authentication spoofing.
debug1: Requesting authentication agent forwarding.
debug1: Remote: No xauth program; cannot forward with spoofing.
X11 forwarding request failed on channel 0
Last login: Mon May 18 16:56:25 2015 from lns-bzn-33-82-252-18-13.adsl.proxad.net
You tried to run a interactive shell.
Sorry, you are not allowed to execute that command.
You are member of the following groups :
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
mga-users mga-shell_access mga-i18n mga-i18n-committers mga-council mga-doc
debug1: channel 0: free: client-session, nchannels 1
Connection to svn.mageia.org closed.
Transferred: sent 3640, received 2600 bytes, in 0.1 seconds
Bytes per second: sent 25618.1, received 18298.7
debug1: Exit status 1
[samageia@localhost ~]$
Comment 2 Pascal Terjan 2015-06-29 20:33:41 CEST
repo software/drakx-installer-help
   RW+ master$    = marja
   RW+ distro/    = marja
   RW+ topic/     = marja
   RW+ refs/tags/ = marja
   RW  master$    = @mga-packagers-committers
   RW  distro/    = @mga-packagers-committers
   RW  topic/     = @mga-packagers-committers
   RW  refs/tags/ = @mga-packagers-committers
   RW  master$    = @mga-i18n-committers
   RW  distro/    = @mga-i18n-committers
   RW  topic/     = @mga-i18n-committers
   RW+ master$    = @mga-sysadmin
   RW+ distro/    = @mga-sysadmin
   RW+ topic/     = @mga-sysadmin
   RW+ refs/tags/ = @mga-sysadmin
   RW+ user/USER/ = @all
   R              = @all

Wondering why there is no + for @mga-i18n-committers and @mga-packagers-committers but no idea what the + means :)

CC: (none) => pterjan

Comment 3 Pascal Terjan 2015-06-29 20:35:22 CEST
R means "read" permission
RW means "read and write", but no rewind
RW+ means "read and write", with rewind allowed
A "rewind" is more often called a "non-fast forward push"; see git docs for what that is. The + was chosen because it is part of the "git push" syntax for non-ff pushes.
Comment 4 Pascal Terjan 2015-06-29 20:45:39 CEST
2015-06-15.12:06:48	8309	ssh	ARGV=lebarhon	SOC=git-receive-pack '/software/drakx-installer-help'	FROM=
2015-06-15.12:06:48	8309		access(software/drakx-installer-help, lebarhon, W, 'any'),-> refs/heads/user/lebarhon/
--
2015-06-15.12:10:29	9442	ssh	ARGV=lebarhon	SOC=git-receive-pack '/software/drakx-installer-help'	FROM=
2015-06-15.12:10:29	9442		access(software/drakx-installer-help, lebarhon, W, 'any'),-> refs/heads/user/lebarhon/

What is being pushed?
Comment 5 Marja Van Waes 2015-06-29 23:41:08 CEST
(In reply to Pascal Terjan from comment #4)
> 2015-06-15.12:06:48	8309	ssh	ARGV=lebarhon	SOC=git-receive-pack
> '/software/drakx-installer-help'	FROM=
> 2015-06-15.12:06:48	8309		access(software/drakx-installer-help, lebarhon, W,
> 'any'),-> refs/heads/user/lebarhon/
> --
> 2015-06-15.12:10:29	9442	ssh	ARGV=lebarhon	SOC=git-receive-pack
> '/software/drakx-installer-help'	FROM=
> 2015-06-15.12:10:29	9442		access(software/drakx-installer-help, lebarhon, W,
> 'any'),-> refs/heads/user/lebarhon/
> 
> What is being pushed?

a new file, the following was done before:

[sam4@localhost fr (master)]$ touch test.txt
[sam4@localhost fr (master)]$ gitk
[sam4@localhost fr (master)]$ gitk
[sam4@localhost fr (master)]$ git add test.txt 
[sam4@localhost fr (master)]$ gitk
[sam4@localhost fr (master)]$ git ci -m "added test.txt for test"
[master a344959] added test.txt for test
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 fr/test.txt
[sam4@localhost fr (master)]$ gitk
[sam4@localhost fr (master)]$ git up
Current branch master is up to date.
[sam4@localhost fr (master)]$ git push

He got a similar
"remote: FATAL: W * lebarhon DENIED by fallthru" error
when trying to push a change to an existing file in
http://gitweb.mageia.org/software/i18n/tools/tree/docs/stylesheets
(many weeks ago)
Comment 6 Pascal Terjan 2015-06-29 23:49:48 CEST
What surprises me in the log is to see refs/heads/user/lebarhon/ instead of refs/heads/master which is what I mean by "What is being pushed".

Do you know where refs/heads/user/lebarhon/ comes from?
Comment 7 Pascal Terjan 2015-06-29 23:51:43 CEST
Complete log of one attempt:

2015-06-15.12:10:29     9442            access(software/drakx-installer-help, lebarhon, W, 'any'),-> refs/heads/user/lebarhon/
2015-06-15.12:10:29     9442            trigger,Writable,access_1,ACCESS_1,software/drakx-installer-help,lebarhon,W,any,refs/heads/user/lebarhon/
2015-06-15.12:10:29     9442    pre_git software/drakx-installer-help   lebarhon        W       any     -> refs/heads/user/lebarhon/
2015-06-15.12:10:29     9442            system,git,shell,-c,git-receive-pack '/var/lib/git/repositories/software/drakx-installer-help.git'
2015-06-15.12:10:29     9442            update,software/drakx-installer-help,lebarhon,W,refs/heads/master,e7622bd2e0121890520d78c66192707287df29b3,a344959a8e97170d68c4896c1088b84d78bf1b07
2015-06-15.12:10:29     9442    die     W refs/heads/master software/drakx-installer-help lebarhon DENIED by fallthru
2015-06-15.12:10:29     9442    END
Comment 8 Marja Van Waes 2015-06-30 00:08:01 CEST
(In reply to Pascal Terjan from comment #6)
> What surprises me in the log is to see refs/heads/user/lebarhon/ instead of
> refs/heads/master which is what I mean by "What is being pushed".
> 
> Do you know where refs/heads/user/lebarhon/ comes from?

I don't have the slightest idea

All I know is that, when sitting next to him in Lille (a weekend in May), that he did repeatedly fetch a fresh clone from software/i18n/tools/, make a change to a file, do 'git add <file>', 'git ci "<message>"' and 'git push'... all the same as I do it.

I cannot imagine he started adding some funny step
Comment 9 Marja Van Waes 2015-06-30 00:17:53 CEST
(In reply to Pascal Terjan from comment #2)
> repo software/drakx-installer-help

mga-doc members should have access, and be able to tag, too

Docteam creates the html help files in Calenco, and then pushes them to software/drakx-installer-help
Comment 10 Pascal Terjan 2015-06-30 00:45:04 CEST
My impression from the attempts I found in logs is that some strange branching happened.

How is the clone done?
Comment 11 Pascal Terjan 2015-06-30 00:52:37 CEST
(It could also be something in ~/.gitconfig)
Comment 12 Marja Van Waes 2015-06-30 12:00:16 CEST
Thanks, Pascal.

I may have forgotten to check his ~/.gitconfig :-/

@ lebarhon

Do you mind attaching your ~/.gitconfig
(after checking it still has a date from before your last try

I can't imagine you started cloning in a different way than

git clone mga:software/i18n/tools and
git clone mga:software/drakx-installer-help

but please say so if you did ;-)
Comment 13 André DESMOTTES 2015-06-30 12:36:49 CEST
(In reply to Marja van Waes from comment #12)
> Thanks, Pascal.
> 
> I may have forgotten to check his ~/.gitconfig :-/
> 
> @ lebarhon
> 
> Do you mind attaching your ~/.gitconfig
> (after checking it still has a date from before your last try

Hereafter the one that is on the laptop we had in Lille, its date is 2015/May 16, it is the Saturday we were in Lille.

[user]
     name = lebarhon
     email = lebarhon@free.fr
[color]
     ui = true
[url "git://git.mageia.org/"]
     insteadof = mga:
[url "ssh://git@git.mageia.org/"]
     pushInsteadof = mga:
[alias]
     ci = commit
     up = pull --rebase
[push]
     default = matching

Hereafter the one that is on my desktop which have exactly the same behaviour,

 its date is 2013 october

[user]
     name = lebarhon
     email = lebarhon@free.fr
[color]
     ui = true
[url "git://git.mageia.org/"]
     insteadof = mga:
[url "ssh://git@git.mageia.org/"]
     pushInsteadof = mga:
[alias]
     ci = commit
     up = pull --rebase
[push]
     default = matching
[gui]
	recentrepo = /home/samageia/Mageia/software

> 
> I can't imagine you started cloning in a different way than
> 
> git clone mga:software/i18n/tools and
> git clone mga:software/drakx-installer-help
> 
> but please say so if you did ;-)

no, I didn't !
Comment 14 Marja Van Waes 2015-06-30 14:39:09 CEST
The one from Lille looks OK 

Btw, back then we also had general connection problems and later it turned out that updating your public ssh key had not worked.

Did you try again with the "Lille" laptop after Neoclust updated your ssh key?

The other one contains something of which I do not understand what it does, nor where it comes from:

  [gui]
	  recentrepo = /home/samageia/Mageia/software


If pushing to git with the laptop from Lille works, then I'd say: try removing those lines. But if the Lille laptop still gives the "remote: FATAL..." error, too, then that's useless :-/
Comment 15 Pascal Terjan 2015-06-30 14:40:27 CEST
The extra lines were probably added by some gui (gitk?) I don't think they can be related.
I am out of ideas :(
Comment 16 Marja Van Waes 2015-06-30 15:13:41 CEST
(In reply to Pascal Terjan from comment #15)
> The extra lines were probably added by some gui (gitk?) I don't think they
> can be related.
> I am out of ideas :(

Thanks a lot for all your effort.

@ Colin

Neoclust ran out of ideas, too, so hoping you have some time.

Should we ask lebarhon to create a new user in identity, and then open a new bug report to add the new him to all the groups he belongs to, to see whether that solves the problem, or is there something else that could still be tried?

Assigning to you, because I vaguely remember you won't read these lines when you're in the CC ;-)

Assignee: sysadmin-bugs => mageia

Comment 17 Pascal Terjan 2015-06-30 15:15:44 CEST
I could try some other things like adding my key to his account and trying to reproduce.
Comment 18 Pascal Terjan 2015-06-30 15:23:06 CEST
I just changed ~git/.ssh/authorized_keys to commit as him instead of me when using my key, and it worked, so this definitely looks like a client problem.

pterjan@pterjan:/tmp$ git clone git://git.mageia.org/software/drakx-installer-help
Cloning into 'drakx-installer-help'...
remote: Counting objects: 4302, done.
remote: Compressing objects: 100% (1304/1304), done.
remote: Total 4302 (delta 3750), reused 3476 (delta 2996)
Receiving objects: 100% (4302/4302), 5.11 MiB | 719.00 KiB/s, done.
Resolving deltas: 100% (3750/3750), done.
Checking connectivity... done.
pterjan@pterjan:/tmp$ cd drakx-installer-help
pterjan@pterjan:/tmp/drakx-installer-help$ vim README
pterjan@pterjan:/tmp/drakx-installer-help$ git commit -a -m 'Remove an empty line'
[master 43ec262] Remove an empty line
 1 file changed, 1 deletion(-)
pterjan@pterjan:/tmp/drakx-installer-help$ git push ssh://git@git.mageia.org/software/drakx-installer-help
Enter passphrase for key '/home/pterjan/.ssh/id_rsa-ext': 
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 297 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Sending notification emails to: soft-commits@ml.mageia.org
To ssh://git@git.mageia.org/software/drakx-installer-help
   305f535..43ec262  master -> master
Comment 19 Marja Van Waes 2015-06-30 15:38:04 CEST
(In reply to Pascal Terjan from comment #18)
> I just changed ~git/.ssh/authorized_keys to commit as him instead of me when
> using my key, and it worked, so this definitely looks like a client problem.
> 

Brilliant idea :-)

Now to figure out what's wrong on client side :-/

Did you restore his key?
Comment 20 Pascal Terjan 2015-06-30 15:41:46 CEST
I restored mine :)

The way it works, as everyone connect as git user, is that based on the key it will run a command (set in ~git/.ssh/authorized_keys), so I changed the command associated to my key to commit as him instead of me, and didn't touch the one associated with his key.
Comment 21 Marja Van Waes 2015-06-30 15:52:54 CEST
(In reply to Pascal Terjan from comment #20)
> I restored mine :)
> 
> The way it works, as everyone connect as git user, is that based on the key
> it will run a command (set in ~git/.ssh/authorized_keys), so I changed the
> command associated to my key to commit as him instead of me, and didn't
> touch the one associated with his key.

Thanks :-D

@ lebarhon

Could you please try one more time? Just to rule out that Pascal committing as you did somehow magically clear something out, or that it got fixed in between your last attempt and Pascal's succesful attempt?

If that doesn't work, then I can't think of anything ... I think you already tried clean install, different users and such :-/
Comment 22 Marja Van Waes 2015-06-30 16:00:14 CEST
Btw, is there only one blacklist, for all of mageia.org, or are there several ones?
Comment 23 Colin Guthrie 2015-06-30 17:35:53 CEST
I would like to see the "git remote -v" and "git branch" output and full output from the failing "git push" too (including the commands executed).

I suspect something has gone wrong with the clone (i.e. at the client side) too and ultimately just reporting the problem via the "Marja proxy" is likely losing critical information!
Comment 24 André DESMOTTES 2015-07-01 10:00:11 CEST
(In reply to Marja van Waes from comment #21)
> 
> @ lebarhon
> 
> Could you please try one more time? Just to rule out that Pascal committing
> as you did somehow magically clear something out, or that it got fixed in
> between your last attempt and Pascal's succesful attempt?

This time, there is something new:

[samageia@localhost git]$ git clone mga:software/i18n/tools
Clonage dans 'tools'...
remote: Counting objects: 11256, done.
remote: Compressing objects: 100% (3420/3420), done.
remote: Total 11256 (delta 8766), reused 10249 (delta 7787)
Réception d'objets: 100% (11256/11256), 8.11 MiB | 1.42 MiB/s, done.
Résolution des deltas: 100% (8766/8766), done.
Vérification de la connectivité... fait.
[samageia@localhost git]$ cd tools
[samageia@localhost tools]$ vim readme
[samageia@localhost tools]$ vim README
[samageia@localhost tools]$ gitk
[samageia@localhost tools]$ git add README
[samageia@localhost tools]$ gitk
[samageia@localhost tools]$ git ci -m "empty line added for test"
[master df35bad] empty line added for test
 1 file changed, 1 insertion(+)
[samageia@localhost tools]$ gitk
[samageia@localhost tools]$ git up
La branche courante master est à jour.
[samageia@localhost tools]$ git push
X11 forwarding request failed on channel 0
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 287 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Sending notification emails to: soft-commits@ml.mageia.org
To ssh://git@git.mageia.org/software/i18n/tools
   fd3c55b..df35bad  master -> master
[samageia@localhost tools]$
Comment 25 Colin Guthrie 2015-07-01 10:03:04 CEST
(In reply to André DESMOTTES from comment #24)
> This time, there is something new:

> [samageia@localhost tools]$ git push
> X11 forwarding request failed on channel 0
> Counting objects: 5, done.
> Delta compression using up to 4 threads.
> Compressing objects: 100% (3/3), done.
> Writing objects: 100% (3/3), 287 bytes | 0 bytes/s, done.
> Total 3 (delta 2), reused 0 (delta 0)
> remote: Sending notification emails to: soft-commits@ml.mageia.org
> To ssh://git@git.mageia.org/software/i18n/tools
>    fd3c55b..df35bad  master -> master
> [samageia@localhost tools]$

So a fresh clone + edit + push works fine. The problem is definitely in our other clone and a client-side problem.

Status: NEW => RESOLVED
Resolution: (none) => INVALID

Comment 26 Marja Van Waes 2015-07-01 12:58:39 CEST
@ André

Was there anything that had changed on your system when doing this last fresh clone, that might have helped?

If so, and if a sysadmin confirms it's related, it would probably be good to add that to our wiki instructions about using git.
Comment 27 André DESMOTTES 2015-07-01 15:27:14 CEST
(In reply to Marja van Waes from comment #26)
> @ André
> 
> Was there anything that had changed on your system when doing this last
> fresh clone, that might have helped?
> 

Nothing at all, I redid the test from my laptop (the one I had in Lille, the first test -comment 24- was done from my desktop) and it is OK too. If something changed, it changed the same way on both PCs.
The drawback when a problem isn't clearly authenticated is that it may re-appear at any time.
We will see...
Comment 28 Marja Van Waes 2015-07-01 20:41:27 CEST
(In reply to André DESMOTTES from comment #27)

> The drawback when a problem isn't clearly authenticated is that it may
> re-appear at any time.
> We will see...

Next time this happens to someone, we should ask him to try again with a fresh checkout at least once a week.

Maybe a Valstar reboot (after the problems in DC) solved it? Or an update of I-don't-know-which-package on your systems?

Anyway, thanks to neoclust, pterjan and colin for all their help!
Comment 29 Marja Van Waes 2015-07-15 13:01:17 CEST
(In reply to Colin Guthrie from comment #25)

> 
> So a fresh clone + edit + push works fine. The problem is definitely in our
> other clone and a client-side problem.

Sorry, dear Colin, but it keeps nagging me that this bug was closed as invalid.

Yes, I did make a mistake by pinging about this bug on the ml without first asking lebarhon to test whether it was still valid

However, a fresh clone + edit + push was tried very many times (e.g. when neoclust asked for it when he looked into this issue, and e.g. when filip asked to try it on something different than on software/i18n/tools)

Lebarhon is very, very precise, both in following and in writing and improving instructions. He won't assume and always go by the book.

If he can't get something like this to work, then either the wiki instructions are wrong or incomplete (which would be a bug, too), or there's a bug in needed software, or the problem is on server side.

s/INVALID/FIXED/

Feel free to revert and explain why

Resolution: INVALID => FIXED

Comment 30 Marja Van Waes 2015-07-15 13:02:57 CEST
s/and always go by the book./. He will always go by the book./

Note You need to log in before you can comment on or make changes to this bug.