Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: add ACME (LetsEncrypt, etc) example docs to the standard library #17053

Open
bradfitz opened this issue Sep 10, 2016 · 59 comments
Open

doc: add ACME (LetsEncrypt, etc) example docs to the standard library #17053

bradfitz opened this issue Sep 10, 2016 · 59 comments
Labels
Documentation NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@bradfitz
Copy link
Contributor

Thanks to @crhym3, we now have a suitably-licensed & CLA-clean ACME implementation in https://godoc.org/golang.org/x/crypto/acme (and a high-level package in https://godoc.org/golang.org/x/crypto/acme/autocert).

I'd like to consider privately vendoring that into Go 1.8 and making HTTPS even easier.

I'd like a complete user program with automatic HTTPS certs to look something like:

package main

import (
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", websiteHandler)
    srv := &http.Server{
        Addr:         "example.com:443",
        ACMEServer:   http.LetsEncrypt, // non-empty enables autocert support
        ACMEAgreeTOS: func(tosURL string) bool { return true },
        ACMEEmail:    "foo@bar.com", // (but optional)
    }
    log.Fatal(srv.ListenAndServeTLS("", ""))
}

Misc notes:

  • The ACMEFoo names are placeholders. Maybe they'd deserve their own struct.
  • ACMEServer would be required as the opt-in, and we wouldn't make LetsEncrypt be automatic or preferred, but we would add a constant const LetsEncrypt = "https://acme-v01.api.letsencrypt.org/directory" like the acme package has.
  • ACMEAgreeTOS would be required for now. The ACME protocol requires a TOS agreement because the CAB Forum requires cert issuers to have a legal relationship with the people getting certs or something. It's mostly a formality, but we shouldn't make it automatically say "yes" either, even though I don't think LetsEncrypt themselves care. Maybe we could export the https://godoc.org/golang.org/x/crypto/acme#AcceptTOS func in the net/http package to reduce the boilerplate.
  • ACMEEmail is optional. If provided, your ACME cert provider can keep you updated on problems or changes.
  • the default cache directory would be automatic. We could provide a string to let people pick an alternate directory. If you want to do something more complicated (e.g. cache coordination over a cluster), then you can just import the golang.org/x/crypto/acme/autocert package yourself. This is analogous to the HTTP/2 situation where common HTTP/2 is provided automatically, but weird uses require importing the guts.
  • srv.ListenAndServeTLS("", "") is already valid for the past few releases, since 6a208ef for net/http: Server.ListenAndServeTLS not compatible with tls.Config.GetCertificate #14268 requested by @willchan specifically for LetsEncrypt stuff. It's a little ugly but works. Maybe we could provide instead a new method or option which also listens on an cleartext port 80 and redirects HTTP to HTTPS, optionally with a HSTS header.

My goal is for HTTPS to be dead simple out of the box.

Thoughts?

/cc @adg @broady @campoy @quentinmit @rsc @robpike @ianlancetaylor @mholt @crhym3

@bradfitz bradfitz added this to the Proposal milestone Sep 10, 2016
@bradfitz bradfitz self-assigned this Sep 10, 2016
@mholt
Copy link

mholt commented Sep 10, 2016

Like what I'm seeing so far!

I wonder if the ACME configuration should be in a separate struct value -- do we want to tether the http.Server type to ACME concretely?

One of the requests we've had in Caddy is to abstract the way certificates are Obtain()ed and Renew()ed -- in other words, an interface with approximately these two methods. An ACME client would be one implementation, a hashicorp/vault implementation might be another, etc. So I wonder if, instead, there should be some sort of interface type (TLSManager? Not sure what you'd call it), which some type tls.ACMEClient implements (or something like that).

In fact, thinking on these lines, it might be beneficial to drop down to the tls package somehow when working with ACME. In other words, you create a TLS listener that has some notion of ACME, but the HTTP server doesn't care. It's just one application of TLS and uses that listener and goes about its usual business.

FWIW, In Caddy I found myself writing essentially a wrapper type over tls.Config that specifies all the ACME stuff and makes the GetCertificate callback, etc. It all happens within TLS (not just HTTPS), so how about an easy way to make an ACME-capable TLS listener that you pass into http.Serve()?

@nhooyr
Copy link
Contributor

nhooyr commented Sep 10, 2016

I've been moving all of my Go servers to use ACME (with letsencrypt) and I've been using Russ's letsencrypt package for now because it is so simple and easy. However, I've also contributed to the acme package that @crhym3 wrote because its design was inspired by Russ's package and because I wanted ACME to be in the stdlib. I think it is a very common use case that we should cater towards.

However, I don't like the use of fields in http.Server to configure ACME. It's very easy to just create a tls.Listener using acme.Manager's GetCertificate callback. However for convenience, it just makes more sense for this to be part of crypto/tls rather than purely net/http so that even generic TLS servers can use ACME more easily.

(as I was writing this, I noticed @mholt's comment which I 👍)

@rolandshoemaker
Copy link
Member

This is super cool!

With my Let's Encrypt hat on: One important thing to consider though is that currently neither the ACME spec nor the API implementation provided by Let's Encrypt should be considered completely stable. The Let's Encrypt API currently implements a amalgam of the four ACME RFC drafts which we colloquially refer to as v01. Once the RFC is finalized we intend to implement a new API version, v02, which will be a strict implementation of the final specification language which will not be completely backwards compatible with v01.

Once v02 is made public (we are currently aiming for somewhere around the start of 2017) we will provide a timetable for depreciating the v01 implementation . Given this might it make sense to wait until the finalized stable API is available before adding this support to the stdlib in order to prevent churn and breakage in users who don't update?

@tmornini
Copy link

My goal is for HTTPS to be dead simple out of the box

👍🏻

@bradfitz
Copy link
Contributor Author

Once v02 is made public (we are currently aiming for somewhere around the start of 2017) we will provide a timetable for depreciating the v01 implementation . Given this might it make sense to wait until the finalized stable API is available before adding this support to the stdlib in order to prevent churn and breakage in users who don't update?

I totally agree. But Go 1.8 isn't due until 2017 anyway. The timing might work out? But even Go 1.9 is acceptable, if that works out best timing-wse.

@jsha
Copy link

jsha commented Sep 10, 2016

Another Let's Encrypt developer here. Wrote out a post saying what @rolandshoemaker said about upcoming ACME changes, but he beat me to it!

Also, ideally the design should encourage disk-based caching as much as possible. If Go programs are designed to issue on each startup, people will very quickly run into Let's Encrypt rate limits. I see that autocert already has a Cache, so it's probably enough to include the Cache mechanism in the library and make sure the examples use it.

Lastly: This is great, and I'm super happy to see it. This is exactly the type of thing Let's Encrypt exists to enable. Thanks!

@bradfitz
Copy link
Contributor Author

@jsha, as I said in my first post, caching would automatic. There wouldn't even be an option to disable it. Only an option to change the directory it uses.

@jsha
Copy link

jsha commented Sep 10, 2016

Got it! I read too quickly. :-)

@dlsniper
Copy link
Contributor

dlsniper commented Sep 10, 2016

As awesome as this sounds, I think this shouldn't be included in the standard library.

I understand that people will say: it's about security, or other arguments, which, yes, are valid.

But please consider this:

  • the ACME standards are in in Draft right now
  • the functionality exists outside of the standard library just fine

Considering proposals to include different useful things in the language or standard library have been shut down of over the course of years, I would like to know if this means a change in vision on how the standard library / language should evolve from here on out. Does it mean that I can raise a proposal for having ? : syntax in the language? It is super useful in so many cases but it has been deemed not necessary as we can do if / else.

@bradfitz
Copy link
Contributor Author

@mholt, @nhooyr, as I wrote:

The ACMEFoo names are placeholders. Maybe they'd deserve their own struct.

I agree they're probably too stuttery. Being a listener is interesting, though. Maybe:

package main

import (
    "crypto/tls"
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", websiteHandler)
    var srv http.Server
    log.Fatal(srv.Serve(&tls.ACMEListener{
        Domain:   "foo.com",
        Provider: tls.LetsEncrypt,
        AgreeTOS: tls.YesIAgreeTOS,
    }))
}

@nhooyr
Copy link
Contributor

nhooyr commented Sep 10, 2016

@bradfitz I like that. But now, I'm not sure if crypto/tls is the best place for such a listener. Perhaps we should add acme.Listener?

@bradfitz
Copy link
Contributor Author

bradfitz commented Sep 10, 2016

@dlsniper, the timing with regards to its draft status is discussed above. See comments from @rolandshoemaker and @jsha.

the functionality exists outside of the standard library just fine

Yes. And maybe all we do here is add documentation examples on how to use autocert. Maybe that's enough. I'd like it to be even easier, though.

Does it mean that I can raise a proposal for having ? : syntax in the language?

You can propose anything you like, but that discussion happened a long time ago and was already decided. If you insist on raising it again, please do it elsewhere and not in this issue.

@bradfitz
Copy link
Contributor Author

@nhooyr, I don't really want to add a new package, though.

We already have https://golang.org/pkg/net/http/httputil/ though. Maybe that's a suitable location.

@mholt
Copy link

mholt commented Sep 10, 2016

@bradfitz That's better. I imagine the tls.ACMEListener will listen only on port 443?

And maybe the Provider value could use a sensible default. Let's Encrypt's endpoint is a good one -- the question is, should it be their staging endpoint or production one? If people are testing their Go program against the "default" endpoint which happens to be the production endpoint, especially if they're wiping their file system (containers?) or cleaning a test directory or something, it could hit CA rate limits pretty fast. So maybe, on second thought, they should be required to specify an endpoint either way...

And why not just true for agree TOS?

@nhooyr
Copy link
Contributor

nhooyr commented Sep 10, 2016

@bradfitz actually, I meant the high level package https://godoc.org/golang.org/x/crypto/acme/autocert.

@nhooyr
Copy link
Contributor

nhooyr commented Sep 10, 2016

So

package main

import (
    "crypto/acme"
    "crypto/acme/autocert"
    "crypto/tls"
    "log"
    "net/http"
)

func main() {
    http.HandleFunc("/", websiteHandler)
    var srv http.Server
    log.Fatal(srv.Serve(&autocert.Listener{
        Domain:   "foo.com",
        Provider: acme.LetsEncrypt,
        AgreeTOS: autocert.AcceptTOS,
    }))
}

I think it would fit in with autocert being the high level package for using acme.

@bradfitz
Copy link
Contributor Author

@mholt, I addressed my opinion on the default in my initial proposal text at top:

ACMEServer would be required as the opt-in, and we wouldn't make LetsEncrypt be automatic or preferred, but we would add a constant const LetsEncrypt = "https://acme-v01.api.letsencrypt.org/directory" like the acme package has.

We could provide two LetsEncrypt constants for prod vs staging but I don't want to make LetsEncrypt be automatic. Depending on any specific organization (for-profit or otherwise) rubs me the wrong way. I want users to explicitly opt-in to it and know what's happening. (Or hopefully know what's happening by being curious what the line Provider: acme.LetsEncrypt means)

@nhooyr
Copy link
Contributor

nhooyr commented Sep 10, 2016

@mholt the acme package uses a callback to determine whether or not you agree to the TOS. the autocert package provides a convenient AcceptTOS function that always returns true. Maybe we could just remove it? I think pretty much everyone just agrees, even if they don't read it. For people who actually care, they can use autocert.

@mholt
Copy link

mholt commented Sep 10, 2016

@nhooyr

I think it would fit in with autocert being the high level package for using acme.

The only issue there is that you then have to import an external package to use it.

@bradfitz
Copy link
Contributor Author

bradfitz commented Sep 10, 2016

@nhooyr, you can basically already do that. This proposal is about whether Go should come out of the box with ACME support, now that ACME is a thing. Go has always been very pro-easy-HTTPS, but ACME wasn't an option at the time when Go first came out.

The question is whether we continue the tradition of promoting easy HTTPS.

@bradfitz
Copy link
Contributor Author

@nhooyr,

Maybe we could just remove it? I think pretty much everyone just agrees, even if they don't read it.

I already addressed that. See my original text at the top of this issue.

@mholt
Copy link

mholt commented Sep 10, 2016

@bradfitz

I addressed my opinion on the default in my initial proposal text at top

Oops, you're right.

Depending on any specific organization (for-profit or otherwise) rubs me the wrong way. ... I want users to explicitly opt-in to it and know what's happening.

This makes sense. And I agree with this philosophy. But if you're going to provide a const for Let's Encrypt, would you have to do that for all public ACME CAs? Maybe the user should just provide their own CA URL string.

@nhooyr
Copy link
Contributor

nhooyr commented Sep 10, 2016

@bradfitz
I understand now. In that case, the current approach with adding it to crypto/tls seems fine to me (though I'd definitely prefer a separate package but I can see why you don't want add more packages to the stdlib).
I was under the impression that the acme package would be added to the standard library which would make it's inclusion in crypto/tls or net/http redundant. I think I remember @crhym3 mentioning it somewhere. I'll try and find it. (edit: never mind I misinterpreted, sorry).

@mholt
I thought Lets Encrypt was the de facto ACME CA? I think it's used enough to warrant it's inclusion as a package level const because I am not even aware of any other public ACME CA.

@bradfitz
Copy link
Contributor Author

But if you're going to provide a const for Let's Encrypt, would you have to do that for all public ACME CAs?

Sure. Or for all public, non-profit (or at least free) ones. I'd rather the user not have to provide a big ugly string. I'd like to keep the boilerplate as tidy as possible.

@bradfitz
Copy link
Contributor Author

bradfitz commented Sep 10, 2016

@nhooyr,

I thought Lets Encrypt was the de facto ACME CA? I think it's used enough to warrant it's inclusion as a package level const because I am not even aware of any other public ACME CA.

Let's Encrypt created the ACME spec, but anybody can implement it. StartCom/StartSSL has announced they plan to use ACME: https://www.ietf.org/mail-archive/web/acme/current/msg01290.html So maybe there will be two. We'll see.

@nhooyr
Copy link
Contributor

nhooyr commented Sep 10, 2016

@bradfitz
Why would we need a second public free CA for ACME? StartCom/StartSSL wants to use it for customers which according to your earlier comment means it wouldn't be included and I agree. For anyone starting another public free ACME CA this seems relevant.

@robpike
Copy link
Contributor

robpike commented Sep 10, 2016

The proposal would dramatically increase the amount of code net/http depends on. For that reason alone, I'd prefer to leave ACME support where it is and settle on clean, standard way to use it. I do not believe it is necessary for it to be in the standard repo for it to be easy to use.

To put it another way, you have stated your goal of easy security, which I applaud, but there has not been sufficient study of the various ways to do that.

@bradfitz
Copy link
Contributor Author

@robpike, it's not much code, if that's the root of your argument.

@gertcuykens
Copy link
Contributor

gertcuykens commented Sep 15, 2016

I am absolutely in favor of this, most common reason somebody would be against it is because of the fear once it's in, you have to live with it for the rest of your life even if LetsEncrypt get replaced by another company that does it differently. I don't understand why we want to turn golang into cobol, can we not be a little more flexible in this go1.x backward compatible stuff please. Meaning if ACME get deprecated, fine, remove it from standard library in next release, change 3 lines of code recompile, done. To me that 1.x promise is more like a curse than a blessing sometimes.

@rsc
Copy link
Contributor

rsc commented Sep 15, 2016

Given that it's not easy today, why not make it easy with an x/ import first and then reevaluate whether it needs the extra step to go into the standard library? I could see an argument for being in the standard library if it were also going to be on by default, but that doesn't seem right anyway. The x/ repos are supposed to be staging for the standard library (like we did with context) but in this case there has been no staging.

@alex
Copy link
Contributor

alex commented Sep 15, 2016

Does Go have a policy about the docs for the standard library including
examples of things in x/ libraries, or even external libraries? It seems
like a great set of first steps is 1) making the API as easy as envisioned
here with an acme.Listener, 2) having the net/http docs examples include
this; neither of which require the Go team to make a long term commitment.

On Thu, Sep 15, 2016 at 2:07 PM, Russ Cox notifications@github.com wrote:

Given that it's not easy today, why not make it easy with an x/ import
first and then reevaluate whether it needs the extra step to go into the
standard library? I could see an argument for being in the standard library
if it were also going to be on by default, but that doesn't seem right
anyway. The x/ repos are supposed to be staging for the standard library
(like we did with context) but in this case there has been no staging.


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#17053 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAADBKmmgsAs9tI_mAzFG-nj_IR3wpMfks5qqYlUgaJpZM4J5z9d
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6

@bradfitz
Copy link
Contributor Author

@rsc, @alex, sounds good. I'll start with the autocert.Listener type and some example docs for std.

@raski
Copy link

raski commented Sep 15, 2016

Stop sending these to me!

torsdag 15 september 2016 skrev Brad Fitzpatrick notifications@github.com:

@rsc https://github.com/rsc, @alex https://github.com/alex, sounds
good. I'll start with the autocert.Listener type and some example docs for
std.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#17053 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACxR0MYNG5u462so5rcGRgvTjxYcjXe9ks5qqY4UgaJpZM4J5z9d
.

@bradfitz
Copy link
Contributor Author

@raski, it looks like you were mentioned on accident above instead of @rasky. You'll have to press the "Unsubscribe" button on the right. This is a Github thing. I can't unsubscribe you.

@glasser
Copy link
Contributor

glasser commented Sep 16, 2016

Allowing the cache implementation (not just directory) to be overridden seems essential to me. Eg, if you are running a binary that compiles this code in on many servers (either your actual app or a custom proxy) which do not share a disk you likely do not want them all to individually and separately talk to LetsEncrypt as that will quickly exhaust your rate limits.

(Otherwise +1 on this general idea. We are a production user of github.com/hlandau/acme but hopefully this library provides similar support, or we can just keep using hlandau.)

@serverhorror
Copy link

Wouldn't it make sense to have something like a security/driver package (along the lines of database/sql/driver) so that one can use e.g. acme or any other provider that has an API and decouple the implementation from the interface? It seems to work well enough for database/sql/driver and there is no actual database driver in the language itself.

Note: I'm not saying that the package shouldn't be included, just that it might be a good idea to have it in a way that makes it possible to swap implementation.

@glasser
Copy link
Contributor

glasser commented Sep 16, 2016

Another argument for a separate listener vs http.Server: presumably you need to configure what domain you're requesting the cert for (vs getting a cert for whatever shows up in a Host header?) In the original snippet I guess the implication was that it derives that from Addr, but it's very common for a server to live behind a proxy and not listen on an interface that directly answers to the domain name for which the cert is needed.

@hlandau
Copy link

hlandau commented Sep 17, 2016

I should also add that if this is done as proposed, any other ACME implementation (and there are many) will almost certainly (and should) use net/http, and thus end up importing this ACME implementation as well. That's rather unpleasant.

@bradfitz
Copy link
Contributor Author

Thanks everyone for the feedback. There's some good stuff in here. However, given the length of this thread and the unlikelihood that anybody's read it all, I think it's time to freeze this thread until it's time for any next step, which won't be at least 4-6 months probably.

The plan at this point is just to implement more in x/crypto/acme and x/crypto/acme/autocert and add more documentation+examples to net/http for Go 1.8.

@golang golang locked and limited conversation to collaborators Sep 17, 2016
@bradfitz bradfitz modified the milestones: Unplanned, Proposal Sep 17, 2016
@bradfitz bradfitz changed the title proposal: add ACME (LetsEncrypt, etc) support to net/http proposal: add ACME (LetsEncrypt, etc) support to the standard library? Sep 17, 2016
@bradfitz bradfitz changed the title proposal: add ACME (LetsEncrypt, etc) support to the standard library? doc: add ACME (LetsEncrypt, etc) example docs to the standard library Oct 3, 2016
@robpike
Copy link
Contributor

robpike commented Jan 3, 2017

Ping @bradfitz - We would like to see the documentation.

@gopherbot
Copy link

CL https://golang.org/cl/39207 mentions this issue.

@ALTree ALTree added NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. and removed Thinking labels Feb 14, 2019
@golang golang unlocked this conversation May 7, 2019
c-expert-zigbee pushed a commit to c-expert-zigbee/crypto_go that referenced this issue Mar 28, 2022
Now users can do 1-line LetsEncrypt HTTPS servers:

    log.Fatal(http.Serve(autocert.NewListener("example.com"), handler))

Updates golang/go#17053

Change-Id: I13fcd3985ebf6bc97a7524cceeb7641cf1b66b22
Reviewed-on: https://go-review.googlesource.com/39207
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alex Vaghin <ddos@google.com>
c-expert-zigbee pushed a commit to c-expert-zigbee/crypto_go that referenced this issue Mar 29, 2022
Now users can do 1-line LetsEncrypt HTTPS servers:

    log.Fatal(http.Serve(autocert.NewListener("example.com"), handler))

Updates golang/go#17053

Change-Id: I13fcd3985ebf6bc97a7524cceeb7641cf1b66b22
Reviewed-on: https://go-review.googlesource.com/39207
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alex Vaghin <ddos@google.com>
LewiGoddard pushed a commit to LewiGoddard/crypto that referenced this issue Feb 16, 2023
Now users can do 1-line LetsEncrypt HTTPS servers:

    log.Fatal(http.Serve(autocert.NewListener("example.com"), handler))

Updates golang/go#17053

Change-Id: I13fcd3985ebf6bc97a7524cceeb7641cf1b66b22
Reviewed-on: https://go-review.googlesource.com/39207
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alex Vaghin <ddos@google.com>
BiiChris pushed a commit to BiiChris/crypto that referenced this issue Sep 15, 2023
Now users can do 1-line LetsEncrypt HTTPS servers:

    log.Fatal(http.Serve(autocert.NewListener("example.com"), handler))

Updates golang/go#17053

Change-Id: I13fcd3985ebf6bc97a7524cceeb7641cf1b66b22
Reviewed-on: https://go-review.googlesource.com/39207
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Alex Vaghin <ddos@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Development

No branches or pull requests