Go io/fs Design (Part I)
As usual, LWN has a good write up on what’s going on in the Go community. This week’s discussion in on the new io/fs
package. The Go team decided to use a Reddit thread to host the conversation about this draft design. LWN points
out that posters raised the following concerns:
- We added status logging by wrapping http.ResponseWriter, and now HTTP/2 push doesn’t work anymore, because our wrapper hides the Push method from the handlers downstream. / It becomes infeasible to use the decorator pattern more
- Doing it “generically” involves a combinatorial explosion of optional interfaces
Ultimately, Russ Cox admits, “It’s true - there’s definitely a tension here between extensions and wrappers. I haven’t seen any perfect solutions for that.”
“What is this all about?” I wondered. so I decided to go spelunking into the world of extension interfaces.
The Proosal
So, first, what is the io/fs proposal? The proposal names a number of interfaces, e.g.,
type FS interface {
Open(name string) (File, error)
}
type File interface {
Stat() (os.FileInfo, error)
Read([]byte) (int, error)
Close() error
}
// A ReadDirFile is a File that implements the ReadDir method for directory reading.
type ReadDirFile interface {
File
ReadDir(n int) ([]os.FileInfo, error)
}
First thing of note is the use of returning an interface. I’ve always questioned the scope of the “accept interfaces, return structs” rule but never found a satisfactory result. Over two years ago, I wrote about an exception to this rule.
I’ve never questioned the utility of interfaces. But I have found justifiable reason for returning an interface. As I wrote back then. I returned an interface because I want to minimize the promise I make to my API users:
I will return something that implements “CombinedOutput()”; It might also implement something else, but I do not make guarantees on anything else.
I’m glad to see a prominent example discussed in public. I don’t want to encourage people to unnecessarily create and return interfaces, but I do want people to think about how much of their API they want to expose.
Extension interfaces and the extension pattern
The proposal takes a break from technicial specifications to talk about the act of naming a pattern.
This ReadDirFile interface is an example of an old Go pattern that we’ve never named before but that we suggest calling an extension interface. An extension interface embeds a base interface and adds one or more extra methods, as a way of specifying optional functionality that may be provided by an instance of the base interface.
Now that we’ve given a name to this pattern, we can have productive conversations about it. Giving a name to a pattern also helps us recognize the pattern when we see it.
“It becomes infeasible to use the decorator pattern”
The io/fs package defines a number of extension interfaces, e.g., ReadFileFS, StatFS, ReadDirFS, and GlobFS.
One of the issues mentioned in the Reddit thread is
We added status logging by wrapping http.ResponseWriter, and now HTTP/2 push doesn’t work anymore, because our wrapper hides the Push method from the handlers downstream.
ResponseWriter is an interface. One example of wrapping this interface is described in blog post by Nick Stogner.
// Since we don’t have direct access to this status code, we will wrap the ResponseWriter that is passed down:
type statusRecorder struct {
http.ResponseWriter
status int
}
func (rec *statusRecorder) WriteHeader(code int) {
rec.status = code
rec.ResponseWriter.WriteHeader(code)
}
func logware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Initialize the status to 200 in case WriteHeader is not called
rec := statusRecorder{w, 200}
next.ServeHTTP(&rec, r)
log.Printf("response status: %v\n", rec.status)
})
}
In Nick’s example, status logging is written in once but any existing http.Handler can take advantage of it by
wrapping a handler with the logware method. This is an example of GoF’s Decorator pattern.
However, some instances of ResponseWriter also implement the extension interface Pusher, which defines
Push(target string, opts *PushOptions) error
The example of the use of Pusher was described in this Go blog:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if pusher, ok := w.(http.Pusher); ok {
// Push is supported.
if err := pusher.Push("/app.js", nil); err != nil {
log.Printf("Failed to push: %v", err)
}
}
// ...
})
Suddenly, the logware approach doesn’t look so great. HTTP/2 push can be broken downstream because statusRecorder
doesn’t implement Push.
Similarly, if you write a FS decorator to cache copies of underlying files in memory, you will end up in a similarly difficult place.
Solutions?
/u/nickcw, the author of RClone, wrote:
This has been a big problem in rclone and I ended up giving up on optional interfaces in the low level interface and going with function pointers - classic vtable style - so that I could see without calling it whether it is supported or not. Sometimes you need to do a lot of setup to call a method and finding out it isn’t supported only when you call it is too late.
/u/nickcw is referring to his use of the Features struct in his code.
// Info provides a read only interface to information about a filesystem.
type Info interface {
// Name of the remote (as passed into NewFs)
Name() string
// Root of the remote (as passed into NewFs)
Root() string
// String returns a description of the FS
String() string
// Precision of the ModTimes in this Fs
Precision() time.Duration
// Returns the supported hash types of the filesystem
Hashes() hash.Set
// Features returns the optional features of this Fs
Features() *Features
}
Where Features look like this:
// Features describe the optional features of the Fs
type Features struct {
// Feature flags, whether Fs
...
// Move src to this remote using server side move operations.
//
// This is stored with the remote path given
//
// It returns the destination Object and a possible error
//
// Will only be called if src.Fs().Name() == f.Name()
//
// If it isn't possible then return fs.ErrorCantMove
Move func(ctx context.Context, src Object, remote string) (Object, error)
...
And used thusly: If the Move function pointer is non-nil, then use it to move an object.
// Move src to this remote using server side move operations.
func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
fs.Debugf(f, "moving obj '%s' -> %s", src, remote)
// if source fs doesn't support move abort
do := f.Fs.Features().Move
if do == nil {
fs.Errorf(src, "source remote (%v) doesn't support Move", src.Fs())
return nil, fs.ErrorCantMove
}
...
do = f.tempFs.Features().Move
...
obj, err := do(ctx, srcObj.Object, remote)
...
}
With /u/nickcw’s design, one can decorate a single method, “Move”, without worrying about breaking other features of the file system.
Why conversations like this is useful
The conversation is still on-going. It may not lead to any changes.
A community is made of humans, with our congitive biases and communicaton problems. Conversations are the only way for a community to bond and grow. This conversation brings to light difficulties of modeling APIs in Go even for experts. IMO it forces us to admit that solutions from an experienced programmers are sometimes no better than those from a novice. Because, sometimes, the problem itself is intrinsically hard.
Related
PS - I subscribed to LWN today. It is a luxury to be able to pay for digital content. It is also important to me to support the few things I really appreciate. I follow a small number of those things. But I support them wholeheartedly.