Create
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Sign up Log in
Products
See all
Community resources
Top groups
Explore all groups
Community resources
Learn
Community resources
Events
Community resources

'git fetch' command failed. stderr: Socket Closed

Klas Mellbourn
Klas Mellbourn April 23, 2013

We have been working for years with TeamCity and for several weeks with git on Stash as version control system. I have set up integration with TeamCity and that has been working fine. We have four different projects that use git

Recently some of the projects have stopped working intermittently.

The build fails to start. Messages like this is shown in red. (with varying agents):

Failed to start build #1094 on Teamcityagent8_Continuous5 (and 1 more)

The build log says very little:

[16:12:46]Checking for changes
[16:12:49]Failed to start build
[16:12:49]Failed to collect changes, error: Error collecting changes for VCS repository 'http://foo-app-sprint:7990/scm/BAR/qux.git#master'
'git fetch' command failed.
stderr: Socket Closed
[16:12:49]Cannot calculate build metric 'percent of class coverage' - no value

Strangely there are other projects that have very similar configuration that work just fine. They use the same build agents.

I have looked at all Stash logs (and event logs), and nothing at all happens in Stash when I force a run in team city. So, it would seem that TeamCity does not even reach the Git server.

If I test the VCS connection in TeamCity it works.

I have tested restarting Stash. It did not help.

We are working with feature branches. But I have tested turning those off and it did not help.

Today the builds have actually worked a couple of times, but the problem returns intermittently.

We are using TeamCity Enterprise 7.1.4 (build 24331) and Atlassian Stash v2.1.2. Both the build agents and Stash run on windows servers.

Any idea on what is wrong and how to solve it?

I have also posted this question on the jetbrains developer forum and serverfault.

Watch
Like Be the first to like this
2178 views

2 answers

Comments for this post are closed

Community moderators have prevented the ability to post new answers.

Post a new question

0 votes
cofarrell
cofarrell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
Get recognized
April 24, 2013

Hi Klas,

Hmm, nothing out of the ordinary there. It's a little worrying that TeamCity are obviously using a SNAPSHOT verison of JGit.

At this it might be nice to see all the logs, just to be sure. If you could create a support zip under Administration > Support > Atlassian Support Tools > Support Zip and click the "Create" button, and raise a support ticket we might be able to help further. It might be good to have the TeamCity logs as well so we can match up when the connections are failing.

I still suspect network connection problems, and might be tempted to run some manual Git fetches from your TeamCity box to Stash. If that fails as well you could enable Git debugging, which may tell you more. I migh also be tempted to fire up Wireshark and see if anything looks odd when the connection fails if you can't reproduce it that way.

Cheers,

Charles

View More Comments
Klas Mellbourn
Klas Mellbourn April 24, 2013

OK, I have created this ticket and uploaded all the logs

https://support.atlassian.com/browse/SSP-1111

Like
David Welsh
David Welsh December 1, 2013

Unable to see SSP-1111, did anything come of the support ticket?

Like
cofarrell
cofarrell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
Get recognized
December 1, 2013

Hi David,

Unfortunately we were unable to get to the bottom of it, in the end Klas decided not to continue with Stash. However, from our end it looked like TeamCity was hitting Stash at an unusually high rate, and Klass was not an admin on that box and could not easily to tweak the settings. Originally they were also running with one CPU on a VM, although upgrading to more didn't appear to help.

From our perspective it's unfortunate we couldn't figure out what was happening, it means we can't use that knowledge if and when it happens again or fix any underlying problems in Stash.

Are you seeing something similar? Are you running TeamCity as well?

Charles

Like
David Welsh
David Welsh December 5, 2013

Hi Charles,

We are using TeamCity (7 Upgrading to 8 Soon) and are working to migrate from our gitolite system (400+ repos) over to Stash.

This error did come up briefly when I was setting up one of our VCS roots to use Stash instead of gitolite but it went away after I triggered the VCS root to re-poll and I haven't seen it since. It did however lead me to search around about it to make sure it wasn't anything too serious.

If it pops up again or becomes a regular thing you'll for sure be seeing a ticket from me as Stash is a big hit with me, and is becoming a big hit with my co-workers.

Cheers,

David

Like
cofarrell
cofarrell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
Get recognized
December 5, 2013

Hey David,

Great to hear Stash is working well for you.

Please let us know if you run into the error again. In general we expect our SCM cache to handle most of the load we see from TeamCity and other CI servers:

https://confluence.atlassian.com/display/STASH/Scaling+Stash+for+Continuous+Integration+performance

Cheers,

Charles

Like
Reply
0 votes
cofarrell
cofarrell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
Get recognized
April 23, 2013

Hi Klas,

Can you see the connection in the atlassian-stash-access.log?

10.32.70.154,10.33.49.85 | http | i790x203966x2 | - | 2013-02-26 13:10:47,346 | "GET /scm/PROJ/repo.git/info/refs HTTP/1.1" | "" "git/1.7.11.3"" | - | - | - | 
10.32.70.154,10.33.49.85 | http | o790x203966x2 | username | 2013-02-26 13:10:47,405 | "GET /scm/PROJ/repo.git/info/refs HTTP/1.1" | "" "git/1.7.11.3"" | - | 59 | dk38dk9 | 
10.32.70.154,10.33.49.85 | http | i790x203966x2 | - | 2013-02-26 13:10:47,346 | "POST /scm/PROJ/repo.git/git-upload-pack HTTP/1.1" | "" "git/1.7.11.3"" | - | - | - | 
10.32.70.154,10.33.49.85 | http | o790x203966x2 | username | 2013-02-26 13:10:47,405 | "POST /scm/PROJ/repo.git/git-upload-pack HTTP/1.1" | "" "git/1.7.11.3"" | fetch | 169 | 8vfp3v |

These correspond to the two requests to the server that are made when fetching. The first GET (which spans two lines) just to determine if there is anything updated, and the second POST downloads any new changes.

Are you running any kind of proxy between the Stash instance and TeamCity?

You may want to raise a support ticket against Stash, so that you can post your logs, but if you're not seeing any access there's is likely something network or TeamCity related.

Cheers,

Charles

View More Comments
Klas Mellbourn
Klas Mellbourn April 23, 2013

Since the problem is intermittent, it is a bit difficult to test, but last I checked I got the impression that nothing was added to the log when the problem occurred.

There should be no proxies between Stash and TeamCity.

Like
Klas Mellbourn
Klas Mellbourn April 23, 2013

Since the problem is intermittent, it is a bit difficult to test, but last I checked I got the impression that nothing was added to the log when the problem occurred.

There should be no proxies between Stash and TeamCity.

Like
Klas Mellbourn
Klas Mellbourn April 23, 2013

Since the problem is intermittent, it is a bit difficult to test, but last I checked I got the impression that nothing was added to the log when the problem occurred.

There should be no proxies between Stash and TeamCity.

Like
cofarrell
cofarrell
Rising Star
Rising Star
Rising Stars are recognized for providing high-quality answers to other users. Rising Stars receive a certificate of achievement and are on the path to becoming Community Leaders.
Get recognized
April 23, 2013

Hi Klas,

I would probably run a test and see what happens by watching the logs (ie tail -f) and forcing another run in TeamCity. Again if you don't see anything on the Stash end, something is likely causing a connection problem.

Sorry I can't be more help. Without knowing more about your environment it's hard to guess what could be causing the connections to be closed.

Cheers,

Charles

Like
Klas Mellbourn
Klas Mellbourn April 23, 2013 
10.64.0.242 | http | o1082x289685x1 | klas.mellbourn | 2013-04-24 18:02:59,423 | "GET /scm/FOO/qux.library.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 624 | 1agkqg | 
10.64.0.242 | http | o1082x289691x4 | klas.mellbourn | 2013-04-24 18:02:59,563 | "GET /scm/FOO/bar.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 515 | npfpm9 | 
10.64.0.242 | http | o1082x289690x3 | klas.mellbourn | 2013-04-24 18:02:59,579 | "GET /scm/FOO/foo.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 593 | xlffrr | 
10.64.0.242 | http | o1082x289693x5 | klas.mellbourn | 2013-04-24 18:02:59,735 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 499 | n6ckn1 | 
10.64.0.242 | http | i1083x289694x1 | - | 2013-04-24 18:03:01,435 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1083x289694x1 | - | 2013-04-24 18:03:01,435 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 0 | - | 
10.64.0.242 | http | i1083x289695x1 | - | 2013-04-24 18:03:01,466 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1083x289695x1 | klas.mellbourn | 2013-04-24 18:03:01,622 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 156 | bg9772 |
Like
Klas Mellbourn
Klas Mellbourn April 23, 2013

When I looked a bit more carefully, I found that there is indeed some logging done to atlassian-stash-access.log when the problem occurrs. Here is what it looked like.

10.64.0.67 | http | o1082x289679x1 | klas.mellbourn | 2013-04-24 18:02:56,646 | "GET /scm/FOO/bar.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 374 | 1y3e28n | 
10.64.0.67 | http | o1082x289681x2 | klas.mellbourn | 2013-04-24 18:02:56,662 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 297 | qnyzue | 
10.64.0.67 | http | o1082x289683x3 | klas.mellbourn | 2013-04-24 18:02:56,708 | "GET /scm/FOO/foo.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 249 | xspxy3 | 
10.64.0.242 | http | i1082x289684x1 | - | 2013-04-24 18:02:58,799 | "GET /scm/FOO/qux.library.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1082x289684x1 | - | 2013-04-24 18:02:58,799 | "GET /scm/FOO/qux.library.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 0 | - | 
10.64.0.242 | http | i1082x289685x1 | - | 2013-04-24 18:02:58,814 | "GET /scm/FOO/qux.library.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | i1082x289686x2 | - | 2013-04-24 18:02:58,877 | "GET /scm/FOO/bar.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1082x289686x2 | - | 2013-04-24 18:02:58,892 | "GET /scm/FOO/bar.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 15 | - |
Like
Klas Mellbourn
Klas Mellbourn April 23, 2013 
10.64.0.242 | http | i1083x289696x1 | - | 2013-04-24 18:03:12,433 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1083x289696x1 | - | 2013-04-24 18:03:12,449 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 16 | - | 
10.64.0.242 | http | i1083x289697x1 | - | 2013-04-24 18:03:12,449 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1083x289697x1 | klas.mellbourn | 2013-04-24 18:03:12,589 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 140 | d4h9qn | 
10.64.0.242 | http | i1083x289698x1 | - | 2013-04-24 18:03:14,367 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1083x289698x1 | - | 2013-04-24 18:03:14,367 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 0 | - | 
10.64.0.242 | http | i1083x289699x1 | - | 2013-04-24 18:03:14,399 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1083x289699x1 | klas.mellbourn | 2013-04-24 18:03:14,617 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 218 | hob0e7 |
(I had to break the log up into several comments due to your strange rule that comments cannot be more than 2000 characters)
Like
Klas Mellbourn
Klas Mellbourn April 23, 2013 
10.64.0.242 | http | i1082x289687x2 | - | 2013-04-24 18:02:58,892 | "GET /scm/FOO/bar.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | i1082x289688x3 | - | 2013-04-24 18:02:58,970 | "GET /scm/FOO/bar.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | i1082x289689x4 | - | 2013-04-24 18:02:58,970 | "GET /scm/FOO/foo.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1082x289688x3 | - | 2013-04-24 18:02:58,970 | "GET /scm/FOO/bar.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 0 | - | 
10.64.0.242 | http | o1082x289689x4 | - | 2013-04-24 18:02:58,970 | "GET /scm/FOO/foo.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 0 | - | 
10.64.0.242 | http | i1082x289690x3 | - | 2013-04-24 18:02:58,986 | "GET /scm/FOO/foo.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | i1082x289691x4 | - | 2013-04-24 18:02:59,048 | "GET /scm/FOO/bar.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | i1082x289692x5 | - | 2013-04-24 18:02:59,236 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1082x289692x5 | - | 2013-04-24 18:02:59,236 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 0 | - | 
10.64.0.242 | http | i1082x289693x5 | - | 2013-04-24 18:02:59,236 | "GET /scm/FOO/ofu.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | - | - | 
10.64.0.242 | http | o1082x289687x2 | tobias.karlsson | 2013-04-24 18:02:59,423 | "GET /scm/FOO/bar.git/info/refs HTTP/1.1" | "" "JGit/2.1.0-SNAPSHOT" | - | 531 | 8khet1 |
Like
Reply
Atlassian logo
Answers Developer Questions
TAGS
AUG Leaders

Atlassian Community Events

    See all events