A while ago, Andrew suggested to me that asking about this
at the beginning of October was poor timing, since everyone
was busy with Go 1.4, and that I might have better luck once
general work on 1.5 started. So I'd like to try to reopen this
discussion now.
I'd love to get feedback:
- Do you think that the problem described in the linked
Google doc is really a problem?
- Do you think my suggestion is reasonable?
- Do you have any other ideas?
Thanks, Keith. I agree with you that the go command should do this for Go 1.5, as an experiment guarded by a -vendor flag.
I apologize for the delay in responding. I wanted to let both this discussion and the broader conversation proceed, especially since the community understands and encounters these problems more than we on the Go team at Google do. This has happened: we now have much more information from the community about the problem being solved, both in descriptions and in code, like godep, nut, and gb.
We have been rereading the discussions here and looking at the landscape of both Go vendoring tools and large projects that use them. One fact is clear: people very much want to copy (“vendor”) source code into their projects without modifying import paths. Godep and similar tools do this with GOPATH manipulation, and gb does it by reimplementing the build system. In both cases, the result is not compatible with “go get”: people using standard tools cannot easily install projects managed with tools like gb and godep. The proposal in this thread can help the go command meet these tools halfway, so that, perhaps along with minor changes to gb, godep, and friends, users will be able to “go get” those projects.
Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
This is a limited form of the original proposal in this thread.
In the original proposal (as I understand it), the interpretation of an import depends both on where the source code containing that import sits in the tree and why the package is being compiled. The meaning of an import in package X is different when X is imported by Y than when X is imported by Z. Therefore the compilation would have to happen multiple times. This doesn’t scale and possibly leads to inconsistent constraints.
In the revised proposal, the interpretation of an import depends only on where the source code containing that import sits in the tree.
The revised proposal also uses the word “vendor” instead of “external”, because (1) there is at least one popular vendoring tool (gb) that uses “vendor” and none that we know of that use “external”; (2) “external” sounds like the opposite of “internal”, which is not the right meaning; and (3) in discussions, everyone calls the broader topic vendoring. It would be nice not to bikeshed the name.
As an aside, the terms “internal vendoring” and “external vendoring” have been introduced into some discussions, to make the distinction between systems that rewrite import paths and systems that do not. With the addition of vendor directories to the go command, we hope that this distinction will fade into the past. There will just be vendoring.
Example
The gb project ships an example project called gsftp. It has a gsftp program with three dependencies outside the standard library: golang.org/x/crypto/ssh, golang.org/x/crypto/ssh/agent, and github.com/pkg/sftp.
Adjusting that example to use the new vendor directory, the source tree would look like:
$GOPATH
| src/
| | github.com/constabulary/example-gsftp/
| | | cmd/
| | | | gsftp/
| | | | | main.go
| | | vendor/
| | | | github.com/pkg/sftp/
| | | | golang.org/x/crypto/ssh/
| | | | | agent/
The file github.com/constabulary/example-gsftp/cmd/gsftp/main.go says:
import (
...
"golang.org/x/crypto/ssh/agent"
)
Because github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh exists and the file being compiled is within the subtree rooted at github.com/constabulary/example-gsftp (the parent of the vendor directory), the source line:
import "golang.org/x/crypto/ssh"
is compiled as if it were:
import "github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh"
(but this longer form is never written).
So the source code in github.com/constabulary/example-gsftp depends on the vendored copy of golang.org/x/crypto/ssh, not one elsewhere in $GOPATH.
In this example, all the dependencies needed by gsftp are (recursively) supplied in the vendor directory, so “go install” does not read any source files outside the gsftp Git checkout. Therefore the gsftp build is reproducible given only the content of the gsftp Git repo and not any other code. And the dependencies need not be edited when copying them into the gsftp repo. And potential users can run “go get github.com/constabulary/example-gsftp/cmd/gsftp” without needing to have an additional vendoring tool installed or special GOPATH configuration.
The point is that adding just the vendor directory mechanism to the go command allows other tools to achieve their goals of reproducible builds and not modifying vendored source code while still remaining compatible with plain “go get”.
Discussion
There are a few points to note about this.
The first, most obvious, and most serious is that the resolution of an import must now take into account the location where that import path was found. This is a fundamental implication of supporting vendoring that does not modify source code. However, the resolution of an import already needed to take into account the current GOPATH setting, so import paths were never absolute. This proposal allows the Go community to move from builds that require custom GOPATH configuration beforehand to builds that just work, because the (more limited) configuration is inferred from the conventions of the source file tree. This approach is also in keeping with the rest of the go command.
The second is that this does not attempt to solve the problem of vendoring resulting in multiple copies of a package being linked into a single binary. Sometimes having multiple copies of a library is not a problem; sometimes it is. At least for now, it doesn’t seem that the go command should be in charge of policing or solving that problem.
The final point is that existing tools like godep, nut, and gb will need to change their file tree layouts if they want to take advantage of compatibility with “go get”. However, compatibility with “go get” used to be impossible. Also, combined with eventual agreement on the vendor-spec, it should be possible for the tools themselves to interoperate.
Deployment
The signals from the Go community are clear: the standard go command must support building source trees in which dependencies have been vendored (copied) without modifications to import paths.
We are well into the Go 1.5 cycle, so caution is warranted. If we put off making any changes, then vendoring tools and “go get” will remain incompatible for the next eight months. On the other hand, if we can make a small, targeted change, then vendoring tools can spend the next eight months experimenting and innovating and possibly converging on a common file tree layout compatible with the go command.
We believe that the experimental proposal above is that small, targeted change. It seems to be the minimal adjustment necessary to support fetching and building vendored, unmodified source code with “go get”. There are many possible extensions or complications we might consider, but for Go 1.5 we want to do as little as possible while remaining useful.
The use of an explicit -vendor flag should help contain the experiment now and ease the transition later. For Go 1.5, users who want to participate in the experiment can opt in by using “go get -vendor”, “go install -vendor”, and so on.
The new semantics changes the meaning of (breaks) source trees containing directories already named “vendor”. Of the over 60,000 listed on godoc.org, there are fewer than 50 such examples. Putting the new semantics behind the -vendor flag avoids breaking those trees for now.
If we decide that the -vendor behavior is correct, then in a later release (possibly Go 1.6) we would make the -vendor flag default to true. Projects containing “vendor” directories could still use “-vendor=false” to get the old behavior while they convert their code. In a still later release (possibly Go 1.7) we would remove the -vendor flag, locking in the vendoring semantics.
--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Sorry, accidentally sent off list:
As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is.
This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.
Toward this end, many users use goimports, which automatically orders and formats imports. Could goimports help legibility by organizing the imports?
import (
“fmt”
“log”
// vendored packages (comment could be explicit or not)
“github.com/pkg/sftp”
“github.com/x/crypto/ssh”
// non-vendored
sftp2 “github.com/pkg/sftp”
“github.com/gonum/floats”
)
Why is this style of vendoring chosen? I don’t wish to bike shed, but this proposal seems to take per-package vendoring as a given and discusses how that should happen. For example, why not specify a specific version of a package (by commit hash or something), and then have a package with that hash. Then there could be a $GOPATH/vendor that contains all of the vendored packages. This would scale better when there is a single vendored package imported by many other packages, and could help (though not solve) the duplicitous-import issue.
--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/74zjMON9glU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
Unrelated to this change, but I'd like to suggest a small change. How about introducing the -x. prefix for flags that are experimental? It should allow you to flag when things are experimental, just like golang.org/x/ and still be able to "promote" them to non-experimental later on. For this example, -x.vendor and if it turns out to be a good choice then change it in 1.6 to -vendor?What do you think?
Hi,
Unrelated to this change, but I'd like to suggest a small change. How about introducing the -x. prefix for flags that are experimental? It should allow you to flag when things are experimental, just like golang.org/x/ and still be able to "promote" them to non-experimental later on. For this example, -x.vendor and if it turns out to be a good choice then change it in 1.6 to -vendor?What do you think?
I'm happy with this proposal. [...] Do you foresee modifications to other tools should this experiment go well, such as tools that rename symbols across a GOPATH and oracle?
...
If that is the case then this is insufficient for our needs. Without recursive vendoring or deep rewriting, I don't see how to have two versions of github.com/me/prj that are used for different binaries in a repo.
Moving these things to different git repos would be unpleasant, so I am really saving that as a very last resort.
Tim
If that is the case then this is insufficient for our needs. Without recursive vendoring or deep rewriting, I don't see how to have two versions of github.com/me/prj that are used for different binaries in a repo.
Hmm, then I don't understand the heuristic for where the build looks for a vendor dir. I assumed it was just at the root of the project repo.
Can you clarify that aspect for me? Given a source file $GOPATH/github.com/me/prj/pkg/foo/bar/bar.go, which imports github.com/you/other, what are the places where an vendor/ may exist and be found by the build?
Thanks
Tim
--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/74zjMON9glU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
Ah, so it is recursive. If I vendor a project that has valid vendor/ does then those deps will be preserved, right?
This does imply that a single binary might have multiple copies of the same or different versions of a single dep, but that is for the import tooling to resolve. If the build supports it, the tooling can choose when to use or not use it.
Am I getting it?
Why "vendor" and not "_vendor" ?
Ah, so it is recursive. If I vendor a project that has valid vendor/ does then those deps will be preserved, right?
This does imply that a single binary might have multiple copies of the same or different versions of a single dep, but that is for the import tooling to resolve. If the build supports it, the tooling can choose when to use or not use it.
Am I getting it?
Why "vendor" and not "_vendor" ?
"internal" dirs are still yours, and so you would want them included in things like go test ./... but I would assume one would NOT want vendored deps to have tests run. Doesn't leading underscore have that property?
"internal" dirs are still yours, and so you would want them included in things like go test ./... but I would assume one would NOT want vendored deps to have tests run.
Also, I think a vendor directory in the same directory as a file must be checked
Why do I want to run test under my deps on any regular basis? My own tests are slow enough. If my deps are properly carried, their test results don't change when my code does...
Why do I want to run test under my deps on any regular basis? My own tests are slow enough. If my deps are properly carried, their test results don't change when my code does...
Why do I want to run test under my deps on any regular basis? My own tests are slow enough. If my deps are properly carried, their test results don't change when my code does...
I think _vendor provides the (more) correct semantics out of the box without pushing the extra work to attain correctness on the user and without depending on hypothetical other features.
And it is just as minimal if not more :)
Go chose to make _dir have special meaning, and that special meaning is correct here. I am sorry if people find it ugly take that up with the designers. Most existing vendoring tools do exactly this, and changing it would be silly, IMO of course.
Also, _vendor seems far less likely to collide with legit users of that name today.
--You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
I apologize by proxy for the original design which dictates _whatever is special, but the specialness is warranted for vendored packages. If I 'go test ./...' I do NOT want it kicking off test for vendors deps. Etc.
I apologize by proxy for the original design which dictates _whatever is special, but the specialness is warranted for vendored packages. If I 'go test ./...' I do NOT want it kicking off test for vendors deps. Etc.
We run vendored tests ourselves, NOT as part of every 'go test' . But I am clearly on the losing side of this one, so I will just shut up now.
I agree with Tim. I also suggested _vendor (although I wasn't that insistent)
Hyperbole won't help the situation. The go tool already treats directories starting with an underscore as invisible. Given that property, _vendor is consistent and appropriate.
On Thu, Jun 11, 2015 at 8:17 AM, Brendan Tracey <tracey....@gmail.com> wrote:Sorry, accidentally sent off list:
As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is.As Russ wrote: "However, the resolution of an import already needed to take into account the current GOPATH setting, so import paths were never absolute."That is, a GOPATH can already contain multiple entries. So if GOPATH is /foo:/bar, import "x" might be in /foo/src/x or /bar/src/x.
This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.
Toward this end, many users use goimports, which automatically orders and formats imports. Could goimports help legibility by organizing the imports?
import (
“fmt”
“log”
// vendored packages (comment could be explicit or not)
“github.com/pkg/sftp”
“github.com/x/crypto/ssh”
// non-vendored
sftp2 “github.com/pkg/sftp”
“github.com/gonum/floats”
)I don't foresee users mixing vendored and non-vendored packages. It'll typically be all or none. So I also don't foresee any changes to goimports.
--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
On Jun 12, 2015, at 6:17 AM, Gustavo Niemeyer <gus...@niemeyer.net> wrote:Vendoring and gopkg.in are orthogonal and complementary concepts.Using gopkg.in allows people to provide a public statement of API stability, and also to release and maintain new versions which break said API stability in a convenient and expectable way. Vendoring allows people to lock down the exact code that is being used by an application, and to update the code in a controlled manner.One benefits from API stability even when vendoring, and applications benefit from code lock down even with API stability.
You cannot have paths longer then about 250 on windows (see http://golang.org/issue/3358). Even more, given the way "go test" arranges its files your path will get doubled sometimes. So you really cannot make paths more then about 130.Alex
--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/74zjMON9glU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is. This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.
Toward this end, many users use goimports, which automatically orders and formats imports. Could goimports help legibility by organizing the imports?
Why is this style of vendoring chosen? I don’t wish to bike shed, but this proposal seems to take per-package vendoring as a given and discusses how that should happen. For example, why not specify a specific version of a package (by commit hash or something), and then have a package with that hash. Then there could be a $GOPATH/vendor that contains all of the vendored packages. This would scale better when there is a single vendored package imported by many other packages, and could help (though not solve) the duplicitous-import issue.
Hi Russ,
This looks like a real step forward. It is not clear to me what you intend to happen if I vendor a project that itself has a vendor/ directory with its own deps.
Should it be an error? Or will the dep's deps just be ignored? Or is this truly a recursive definition?
Why "vendor" and not "_vendor" ?
"$GOPATH/src/github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh/agent/..."
It's like the Limbo, how low can you go? Which brings me the my next concern. Perhaps I missed it, but what about the vendored libraries of vendored libraries? Are they going to be yet further down the tree?
"$GOPATH/src/github.com/trans/foo-gstfp/vendor/githubc.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh/agent/..."
Lastly, when you speak of a "small, targeted change", I can't help but read "kludge". Rather than solving the issue full-on it's an attempt just to muddle through so as to effect existing `go` functionality as little as possible.
You cannot have paths longer then about 250 on windows (see http://golang.org/issue/3358). Even more, given the way "go test" arranges its files your path will get doubled sometimes. So you really cannot make paths more then about 130.
Once this gets implemented, what is the need for internal packages? We could just put them in the vendor directory. Using an internal subdirectory may be a convention, but we could get rid of that special case (so, for example, you would import "internal/mypackage", which would be in vendor/internal/mypackage).I am not sure which is the best way to combine both features, but I think that adding both internal and vendor directories with this overlapping of functionality may be a mistake in the long term.
One issue that has not been covered in the spec, suppose you have:b/vendor/foob/vendor/bar/vendor/foo (so pkg bar also imports foo but these two versions of foo might well be different)b/main.gowe probably don't want to import "foo" from the vendored pkg's vendor dir, even if thats longer in path length. That is, vender search path can only contain one level of vendor directory rooted from the pkg itself. Otherwise, this will become very confusing as which vendored "foo" gets used. Accidentally use the vendored pkg's vendored one may not be what the user wants at all.The simple "longest applies" does not seem to address this issue, and I think the spec should cover this.
--
You received this message because you are subscribed to a topic in the Google Groups "golang-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-dev/74zjMON9glU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
On Thu, Jun 11, 2015 at 8:37 PM, Yongjian Xu <i3dm...@gmail.com> wrote:One issue that has not been covered in the spec, suppose you have:b/vendor/foob/vendor/bar/vendor/foo (so pkg bar also imports foo but these two versions of foo might well be different)b/main.gowe probably don't want to import "foo" from the vendored pkg's vendor dir, even if thats longer in path length. That is, vender search path can only contain one level of vendor directory rooted from the pkg itself. Otherwise, this will become very confusing as which vendored "foo" gets used. Accidentally use the vendored pkg's vendored one may not be what the user wants at all.The simple "longest applies" does not seem to address this issue, and I think the spec should cover this.
I parse the meaning of _dir differently than you, but it's not interpretation that matters. I read it as ... will never match.
I think this overall proposal is a step forward and makes the vendoring ecosystem one step closer to consistent.
On Thu, Jun 11, 2015 at 11:33 AM, Tim Hockin <tho...@gmail.com> wrote:Why "vendor" and not "_vendor" ?The most important reason is that we've documented that "[d]irectories beginning with . or _ are ignored by the go tool." That means all the time, not just some of the time.
In addition to what Brad wrote:On Thu, Jun 11, 2015 at 7:04 AM, Brendan Tracey <tracey....@gmail.com> wrote:As you say, this means the resolution of an import now depends on where the path is found. A second implication is that it removes one of my favorite properties about Go; in the current environment I can read an import statement and know exactly where the code is. This issue is clearly very important for the community, which may merit breaking this invariant. I would be sad to see more changes that obscure the link between import path and file location.Yes, absolutely. I agree with you 100%.
We run vendored tests ourselves, NOT as part of every 'go test' . But I am clearly on the losing side of this one, so I will just shut up now.
On Jun 11, 2015 6:39 PM, "Brad Fitzpatrick" <brad...@golang.org> wrote:On Thu, Jun 11, 2015 at 5:58 PM, Tim Hockin <tho...@hockin.org> wrote:I apologize by proxy for the original design which dictates _whatever is special, but the specialness is warranted for vendored packages. If I 'go test ./...' I do NOT want it kicking off test for vendors deps. Etc.
If you update from Go 1.N to Go 1.N+1, you don't want to know whether your vendored dependencies still pass?If a user reports a bug on random $GOOS and you start debugging on that OS, you don't want to know whether your vendored dependencies still pass there?If you really don't want to be bothered with test failures, just instruct your vendoring tool to delete the test files.
Thanks, Keith. I agree with you that the go command should do this for Go 1.5, as an experiment guarded by a -vendor flag.
I apologize for the delay in responding. I wanted to let both this discussion and the broader conversation proceed, especially since the community understands and encounters these problems more than we on the Go team at Google do. This has happened: we now have much more information from the community about the problem being solved, both in descriptions and in code, like godep, nut, and gb.
We have been rereading the discussions here and looking at the landscape of both Go vendoring tools and large projects that use them. One fact is clear: people very much want to copy (“vendor”) source code into their projects without modifying import paths. Godep and similar tools do this with GOPATH manipulation, and gb does it by reimplementing the build system. In both cases, the result is not compatible with “go get”: people using standard tools cannot easily install projects managed with tools like gb and godep. The proposal in this thread can help the go command meet these tools halfway, so that, perhaps along with minor changes to gb, godep, and friends, users will be able to “go get” those projects.
Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
This is a limited form of the original proposal in this thread.
In the original proposal (as I understand it), the interpretation of an import depends both on where the source code containing that import sits in the tree and why the package is being compiled. The meaning of an import in package X is different when X is imported by Y than when X is imported by Z. Therefore the compilation would have to happen multiple times. This doesn’t scale and possibly leads to inconsistent constraints.
In the revised proposal, the interpretation of an import depends only on where the source code containing that import sits in the tree.
The revised proposal also uses the word “vendor” instead of “external”, because (1) there is at least one popular vendoring tool (gb) that uses “vendor” and none that we know of that use “external”; (2) “external” sounds like the opposite of “internal”, which is not the right meaning; and (3) in discussions, everyone calls the broader topic vendoring. It would be nice not to bikeshed the name.
As an aside, the terms “internal vendoring” and “external vendoring” have been introduced into some discussions, to make the distinction between systems that rewrite import paths and systems that do not. With the addition of vendor directories to the go command, we hope that this distinction will fade into the past. There will just be vendoring.
Example
The gb project ships an example project called gsftp. It has a gsftp program with three dependencies outside the standard library: golang.org/x/crypto/ssh, golang.org/x/crypto/ssh/agent, and github.com/pkg/sftp.
Adjusting that example to use the new vendor directory, the source tree would look like:
$GOPATH
| src/
| | github.com/constabulary/example-gsftp/
| | | cmd/
| | | | gsftp/
| | | | | main.go
| | | vendor/
| | | | github.com/pkg/sftp/
| | | | golang.org/x/crypto/ssh/
| | | | | agent/
The file github.com/constabulary/example-gsftp/cmd/gsftp/main.go says:
import (
...
"golang.org/x/crypto/ssh/agent"
)
Because github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh exists and the file being compiled is within the subtree rooted at github.com/constabulary/example-gsftp (the parent of the vendor directory), the source line:
import "golang.org/x/crypto/ssh"
is compiled as if it were:
import "github.com/constabulary/example-gsftp/vendor/golang.org/x/crypto/ssh"
(but this longer form is never written).
So the source code in github.com/constabulary/example-gsftp depends on the vendored copy of golang.org/x/crypto/ssh, not one elsewhere in $GOPATH.
In this example, all the dependencies needed by gsftp are (recursively) supplied in the vendor directory, so “go install” does not read any source files outside the gsftp Git checkout. Therefore the gsftp build is reproducible given only the content of the gsftp Git repo and not any other code. And the dependencies need not be edited when copying them into the gsftp repo. And potential users can run “go get github.com/constabulary/example-gsftp/cmd/gsftp” without needing to have an additional vendoring tool installed or special GOPATH configuration.
The point is that adding just the vendor directory mechanism to the go command allows other tools to achieve their goals of reproducible builds and not modifying vendored source code while still remaining compatible with plain “go get”.
Discussion
There are a few points to note about this.
The first, most obvious, and most serious is that the resolution of an import must now take into account the location where that import path was found. This is a fundamental implication of supporting vendoring that does not modify source code. However, the resolution of an import already needed to take into account the current GOPATH setting, so import paths were never absolute. This proposal allows the Go community to move from builds that require custom GOPATH configuration beforehand to builds that just work, because the (more limited) configuration is inferred from the conventions of the source file tree. This approach is also in keeping with the rest of the go command.
The second is that this does not attempt to solve the problem of vendoring resulting in multiple copies of a package being linked into a single binary. Sometimes having multiple copies of a library is not a problem; sometimes it is. At least for now, it doesn’t seem that the go command should be in charge of policing or solving that problem.
The final point is that existing tools like godep, nut, and gb will need to change their file tree layouts if they want to take advantage of compatibility with “go get”. However, compatibility with “go get” used to be impossible. Also, combined with eventual agreement on the vendor-spec, it should be possible for the tools themselves to interoperate.
Deployment
The signals from the Go community are clear: the standard go command must support building source trees in which dependencies have been vendored (copied) without modifications to import paths.
We are well into the Go 1.5 cycle, so caution is warranted. If we put off making any changes, then vendoring tools and “go get” will remain incompatible for the next eight months. On the other hand, if we can make a small, targeted change, then vendoring tools can spend the next eight months experimenting and innovating and possibly converging on a common file tree layout compatible with the go command.
We believe that the experimental proposal above is that small, targeted change. It seems to be the minimal adjustment necessary to support fetching and building vendored, unmodified source code with “go get”. There are many possible extensions or complications we might consider, but for Go 1.5 we want to do as little as possible while remaining useful.
The use of an explicit -vendor flag should help contain the experiment now and ease the transition later. For Go 1.5, users who want to participate in the experiment can opt in by using “go get -vendor”, “go install -vendor”, and so on.
The new semantics changes the meaning of (breaks) source trees containing directories already named “vendor”. Of the over 60,000 listed on godoc.org, there are fewer than 50 such examples. Putting the new semantics behind the -vendor flag avoids breaking those trees for now.
If we decide that the -vendor behavior is correct, then in a later release (possibly Go 1.6) we would make the -vendor flag default to true. Projects containing “vendor” directories could still use “-vendor=false” to get the old behavior while they convert their code. In a still later release (possibly Go 1.7) we would remove the -vendor flag, locking in the vendoring semantics.
Comments welcome here. I just sent CLs 10922 and 10923 as a prototype.Thanks very much for this proposal.Russ
Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
I think this is exactly the right approach.
I'm happy with this proposal.
From the proposed CLs, it looks like tools that re-write import paths should not have the "/vendor/" in the path. I think that's fine; existing tools don't currently place them there anyway.I think having a flag as currently proposed would allow existing tools to easily opt in. If people wanted to use the go command directly, it might be prudent to support an environment variable as well "GO_VENDOR=1".
If package a imported packages b and c where b and c both vendored a different version of package d what would happen?
"go test ./..." is very convenient. If to test only my own code I must "go test $(go list ./... | grep -v vendor)" or whatever the command is, then I might as well start using a Makefile and "make test" instead.
I'm wondering what would happen if:- I have a package in `$GOPATH/src/myproject` that depends on `somedependency` stored in `$GOPATH/src/myproject/vendor/somedependency`.- I build my project.- I copy a different version of `somedependency` in `$GOPATH/src/somedependency`.- I delete the vendored version in `$GOPATH/src/myproject/vendor/somedependency`.- I rebuilt my project.Will the go tool detect that `somedependency` is stale and rebuild it using the version in `$GOPATH/src/somedependency`?
I caution you to take it slow and think it all through
carefully. There is no need to rush.
As for my own druthers, I much rather see a standard dependency file
that maps an import name to a package name. This way the import name
is always the same regardless of the location -- and it can be nice
and short. Of course, this would give people the freedom to be rather
wonky with import names, but good coders will stick to standards. Then
have a toplevel `ext` directory for the all vendored packages -- no
need for deep directory structures. And note, *vendored packages
should not be checked into the scm!*
Will the go tool detect that `somedependency` is stale and rebuild it using the version in `$GOPATH/src/somedependency`?Yes.
Specifically, we propose that, as an experiment for Go 1.5, we add a temporary “-vendor” flag that causes the go command to add these semantics:
If there is a source directory d/vendor, then, when compiling a source file within the subtree rooted at d, import "p" is interpreted as import "d/vendor/p" if that exists.
When there are multiple possible resolutions, the most specific (longest) path wins.
The short form must always be used: no import path can contain “/vendor/” explicitly.
Import comments are ignored in vendored packages.
Probably godoc will not know about vendoring for Go 1.5. Part of the experiment is learning what tools need to be updated for when we make it official. Godoc is certainly one of them. I expect it would (be made to) get the hyperlinks in import statements correct.
Thanks, Keith. I agree with you that the go command should do this for Go 1.5, as an experiment guarded by a -vendor flag.
--
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
Please feel free to try it out and report problems; note that the Go tree at tip is not terribly stable right now, so it shouldn't be used for building production code. But experimenting with the vendor support should be fine.