Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os: add ReadDir, ReadFile, WriteFile, CreateTemp, MkdirTemp & deprecate io/ioutil #42026

Closed
rsc opened this issue Oct 16, 2020 · 25 comments
Closed

Comments

@rsc
Copy link
Contributor

rsc commented Oct 16, 2020

io/ioutil, like most things with util in the name, has turned out to be a poorly defined and hard to understand collection of things.
As part of #40025, we migrated Discard, NopCloser, and ReadAll to package io.
(They remain in io/ioutil for compatibility, of course, but new code should prefer the ones in io.)
What's left is a few OS file system helpers: ReadDir, ReadFile, TempDir, TempFile, and WriteFile.
I propose we migrate all of these to package os, with a few adjustments.
(Again, they would remain here for compatibility, but new code would prefer the ones in os.)

At that point, io/ioutil would become entirely deprecated.

The signatures would be:

package os

// ReadDir reads the named directory,
// returning all its directory entries sorted by filename.
func ReadDir(name string) ([]DirEntry, error)

// ReadFile reads the named file and returns the contents.
// A successful call returns err == nil, not err == EOF.
// Because ReadFile reads the whole file, it does not treat an EOF from Read
// as an error to be reported.
func ReadFile(name string) ([]byte, error)

// WriteFile writes data to the named file, creating it if necessary.
// If the file does not exist, WriteFile creates it with permissions perm (before umask);
// otherwise WriteFile truncates it before writing, without changing permissions.
func WriteFile(name string, data []byte, perm fs.FileMode) error

// MkdirTemp creates a new temporary directory in the directory dir
// and returns the pathname of the new directory.
// The new directory's name is generated by adding a random string to the end of pattern.
// If pattern includes a "*", the random string replaces the last "*" instead.
// If dir is the empty string, TempFile uses the default directory for temporary files, as returned by TempDir.
// Multiple programs or goroutines calling MkdirTemp simultaneously will not choose the same directory.
// It is the caller's responsibility to remove the directory when it is no longer needed.
func MkdirTemp(dir, pattern string) (string, error)

// CreateTemp creates a new temporary file in the directory dir,
// opens the file for reading and writing, and returns the resulting file.
// The filename is generated by taking pattern and adding a random string to the end.
// If pattern includes a "*", the random string replaces the last "*".
// If dir is the empty string, TempFile uses the default directory for temporary files, as returned by TempDir.
// Multiple programs or goroutines calling CreateTemp simultaneously will not choose the same file.
// The caller can use the file's Name method to find the pathname of the file.
// It is the caller's responsibility to remove the file when it is no longer needed.
func CreateTemp(dir, pattern string) (*File, error)

Notes:

  • os.ReadDir returns []DirEntry, in contrast to ioutil.ReadDir's []FileInfo.
    (Providing a helper that returns []DirEntry is one of the primary motivations for this change.)

  • os.ReadFile is identical to ioutil.ReadFile.

  • os.WriteFile is identical to ioutil.WriteFile.
    I looked into whether the permission bits should be dropped, but Go code in the wild uses a variety of popular settings (for example: 0666, 0644, 0600, 0700, 0777).

  • os.MkdirTemp is identical to ioutil.TempDir.
    It cannot be named os.TempDir because that already exists and returns the default temporary directory.
    MkdirTemp seems like it makes clear that the directory is being created and will appear next to Mkdir in docs.
    I looked into whether the directory argument should be dropped, but about 20% of calls in the wild don't use the empty string.

  • os.CreateTemp is identical to ioutil.TempFile.
    It could be named os.TempFile, but calling it os.CreateTemp is consistent with os.MkdirTemp, makes clear that the file is being created (using TempFile for this would be a function surprisingly different from TempDir), and will appear next to Create in docs.
    I looked into whether the directory argument should be dropped, but about 25% of calls in the wild don't use the empty string.

@rsc rsc added this to the Proposal milestone Oct 16, 2020
@bcmills
Copy link
Contributor

bcmills commented Oct 17, 2020

Would it make sense to put ReadDir, ReadFile, and WriteFile in io/fs instead of os?
It sees like they could plausibly work with any filesystem, not just the os one.

I would argue that they fit naturally with the new Walk API, wherever that ends up: ReadDir, ReadFile, WriteFile, and Walk all share the property that they implement a high-level operation in terms of existing lower-level operations.

@bcmills
Copy link
Contributor

bcmills commented Oct 17, 2020

(MkdirTemp and CreateTemp seem like they belong in os, though: the location of the temporary directory is OS-specific, and CreateTemp, at least, should return a *os.File rather than an abstract fs.File.)

@bcmills
Copy link
Contributor

bcmills commented Oct 17, 2020

Actually, even MkdirTemp and CreateTemp could belong in io/fs, if we define an FS extension interface for a method that returns a default directory for temporary files.

@rsc
Copy link
Contributor Author

rsc commented Oct 20, 2020

@bcmills, io/fs already has ReadDir and ReadFile. It would be fine to add CreateTemp and MkdirTemp if/when io/fs adds any writing operations.

But it probably wouldn't make sense to put them only in io/fs. Reading from the operating system's file systems is insanely common. For example, even if we were starting over we would certainly keep os.Open(name) rather than tell people to use fs.Open(os.Dir("/"), name) or os.Dir("/").Open(name) (neither of which would be correct on Windows anyway).

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Oct 28, 2020
@rsc rsc moved this from Incoming to Active in Proposals (old) Oct 28, 2020
@rsc
Copy link
Contributor Author

rsc commented Oct 29, 2020

Accepting this issue will obsolete #19660, which proposes renaming ioutil.

@gopherbot
Copy link

Change https://golang.org/cl/266365 mentions this issue: all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp

@gopherbot
Copy link

Change https://golang.org/cl/266364 mentions this issue: os: add ReadFile, WriteFile, CreateTemp (was TempFile), MkdirTemp (was TempDir) from io/ioutil

@gopherbot
Copy link

Change https://golang.org/cl/266366 mentions this issue: all: update to use os.ReadDir where appropriate

@rsc
Copy link
Contributor Author

rsc commented Oct 29, 2020

I've posted the CLs that would implement this change in case anyone wants to see what the diffs look like.
I am particularly happy that in the two CLs converting the tree to use the new locations, 217 "io/ioutil" imports were removed but only 51 "os" imports were added, meaning that in ~75% of cases the code was already importing "os" and is simplified by not having to juggle ioutil as well.

@rsc
Copy link
Contributor Author

rsc commented Nov 4, 2020

Based on the discussion, this seems like a likely accept.

@rsc rsc moved this from Active to Likely Accept in Proposals (old) Nov 4, 2020
@earthboundkid
Copy link
Contributor

Getting rid of ioutil is a good conceptual clean up. I know when I first learned Go, I was confused about what was in io vs. ioutil vs. bufio. io itself is a little tricky to understand, but with the cleanup, the explanation is pretty clear. I can imagine writing a blog post for beginners along the lines of:

Package io defines different kinds of "virtual files" either for reading or writing with special features like seeking and closing. io/fs extends the virtual file into a virtual filesystem. In most case, you won't work with io.Readers and io.Writers directly, but either stream them with package bufio or write them to memory with bytes.Buffer or strings.Builder.

The only thing that seems odd given that description is that FileInfo and FileMode are in io/fs instead of io itself. On the one hand, they're part of a "virtual file" so you'd think they'd be in io. OTOH, they only make sense as part of a "virtual filesystem" in which case they should be in io/fs…

@rsc
Copy link
Contributor Author

rsc commented Nov 11, 2020

Instead of "virtual files" say "virtual data streams" and you're all set!

@rsc
Copy link
Contributor Author

rsc commented Nov 11, 2020

No change in consensus, so accepted.

@rsc rsc moved this from Likely Accept to Accepted in Proposals (old) Nov 11, 2020
@rsc rsc changed the title proposal: os: add ReadDir, ReadFile, WriteFile, CreateTemp, MkdirTemp & deprecate io/ioutil os: add ReadDir, ReadFile, WriteFile, CreateTemp, MkdirTemp & deprecate io/ioutil Nov 11, 2020
@rsc rsc modified the milestones: Proposal, Backlog Nov 11, 2020
gopherbot pushed a commit that referenced this issue Dec 2, 2020
…s TempDir) from io/ioutil

io/ioutil was a poorly defined collection of helpers.
Proposal #40025 moved out the generic I/O helpers to io.
This CL for proposal #42026 moves the OS-specific helpers to os,
making the entire io/ioutil package deprecated.

For #42026.

Change-Id: I018bcb2115ef2ff1bc7ca36a9247eda429af21ad
Reviewed-on: https://go-review.googlesource.com/c/go/+/266364
Trust: Russ Cox <rsc@golang.org>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
gopherbot pushed a commit that referenced this issue Dec 9, 2020
…Temp

As part of #42026, these helpers from io/ioutil were moved to os.
(ioutil.TempFile and TempDir became os.CreateTemp and MkdirTemp.)

Update the Go tree to use the preferred names.

As usual, code compiled with the Go 1.4 bootstrap toolchain
and code vendored from other sources is excluded.

ReadDir changes are in a separate CL, because they are not a
simple search and replace.

For #42026.

Change-Id: If318df0216d57e95ea0c4093b89f65e5b0ababb3
Reviewed-on: https://go-review.googlesource.com/c/go/+/266365
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
monogon-bot pushed a commit to monogon-dev/monogon that referenced this issue Nov 22, 2021
The ioutil package has been deprecated in Go 1.16 [1]. This CL removes
all our own users of that package and rewrites them to use their
replacements in the os package. I initially wanted to do this with a
gofix but because all replacements were signature-compatible I just
did it with a few string replaces and then ran goimports to fix up the
imports.

I intentionally didn't rewrite the patches as that would require a
different process and is IMO of less value.

[1] golang/go#42026

Change-Id: Iac6663a1f1ee49f9b1c6e4b3d97e73f2c3b54a13
Reviewed-on: https://review.monogon.dev/c/monogon/+/449
Reviewed-by: Sergiusz Bazanski <serge@monogon.tech>
@gopherbot
Copy link

Change https://go.dev/cl/395918 mentions this issue: io/ioutil: add deprecation markers to ioutil

gopherbot pushed a commit that referenced this issue Mar 28, 2022
All the code in ioutil just forwards functionality to code
in either the io or os packages, per issue 42026.

This change adds the "Deprecated" marker to all the
functions in this package.

For #42026

Fixes #51927

Change-Id: Ia807bc5c0edb06cc80ec7e35917dcfe2ad50f0ea
GitHub-Last-Rev: 3c3603f
GitHub-Pull-Request: #51961
Reviewed-on: https://go-review.googlesource.com/c/go/+/395918
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/399854 mentions this issue: io/ioutil: provide an equivalent for the deprecated ReadDir

gopherbot pushed a commit that referenced this issue Apr 19, 2022
All APIs in the now-deprecated io/ioutil package have a direct
replacement in either the io or os package with the same signature,
with the notable exception of ioutil.ReadDir, as os.ReadDir has a
slightly different signature with fs.DirEntry rather than fs.FileInfo.

New code can easily make use of []fs.DirEntry directly,
but existing code may need to continue using []fs.FileInfo for backwards
compatibility reasons. For instance, I had a bit of code that exposed
the slice as a public API, like:

	return ioutil.ReadDir(name)

It took me a couple of minutes to figure out what the exact equivalent
in terms of os.ReadDir would be, and a code sample would have helped.
Add one for future reference.

For #42026.
For #51927.

Change-Id: I76d46cd7d68fc609c873821755fdcfc299ffd56c
Reviewed-on: https://go-review.googlesource.com/c/go/+/399854
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
ncwsky added a commit to ncwsky/gofound that referenced this issue Aug 10, 2022
使用os包中的ReadFile、ReadDir,详情见golang/go#42026
newpanjing added a commit to sea-team/gofound that referenced this issue Aug 15, 2022
使用os包中的ReadFile、ReadDir,详情见golang/go#42026
chenrui333 added a commit to runatlantis/atlantis that referenced this issue Sep 8, 2022
As of go1.16, io/util package has been deprecated

See issue, golang/go#42026

Signed-off-by: Rui Chen <rui@chenrui.dev>
chenrui333 added a commit to runatlantis/atlantis that referenced this issue Sep 8, 2022
As of go1.16, io/util package has been deprecated

See issue, golang/go#42026

Signed-off-by: Rui Chen <rui@chenrui.dev>

Signed-off-by: Rui Chen <rui@chenrui.dev>
Nabil372 pushed a commit to gocardless/theatre that referenced this issue Nov 15, 2022
Implement go-staticcheck suggestions.

Replaces ioutil with new functions in the os package.
Each function in the os package is a direct replacement for the
previously used functions in the ioutil package.

For more context see these:
- go-critic/go-critic#1019
- golang/go#42026
r4f4 added a commit to r4f4/installer that referenced this issue Nov 18, 2022
`io/ioutil` has been deprecated since go-1.16 [1]. We should use `io`
and `os` instead.

[1] golang/go#42026
r4f4 added a commit to r4f4/installer that referenced this issue Nov 18, 2022
`io/ioutil` has been deprecated since go-1.16 [1]. We should use `io`
and `os` instead.

[1] golang/go#42026
Nabil372 pushed a commit to gocardless/theatre that referenced this issue Nov 30, 2022
Implement go-staticcheck suggestions.

Replaces ioutil with new functions in the os package.
Each function in the os package is a direct replacement for the
previously used functions in the ioutil package.

For more context see these:
- go-critic/go-critic#1019
- golang/go#42026
krrrr38 pushed a commit to krrrr38/atlantis that referenced this issue Dec 16, 2022
As of go1.16, io/util package has been deprecated

See issue, golang/go#42026

Signed-off-by: Rui Chen <rui@chenrui.dev>

Signed-off-by: Rui Chen <rui@chenrui.dev>
hlcfan added a commit to hlcfan/gock that referenced this issue Feb 21, 2023
According to
- golang/go#40025
- golang/go#42026

`ioutil` is deprecated, `io` and `os` packages should be used instead.
hlcfan added a commit to hlcfan/gock that referenced this issue Feb 21, 2023
According to
- golang/go#40025
- golang/go#42026

`ioutil` is deprecated, `io` and `os` packages should be used instead.
@golang golang locked and limited conversation to collaborators Apr 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests

9 participants