nickcw 5 days ago

I used to use interface upgrades in rclone (a program which syncs your data to many different cloud providers) for all the optional features a backend (cloud provider) might have, for example renaming a file.

However the proxy problem became unmanageable. That's when I had a backend (eg the crypt backend which encrypts any other backend) which wraps another backend. Can the crypt backend rename things? That depends on what it is wrapping and you'd have to upgrade the interface call it and then get a special error to find out.

Eventually I switched to a table of bound method pointers which were easy to test against nil to see whether they were implemented and could be filled with a very small amount of reflection.

In my experience interface upgrades are useful but the proxy problem is very real so use sparingly only!

I did suggest at one point (to rsc) an addition to Go which would allow interface proxies to take methods away from their method set at run time to fix this problem, so the proxy could only have the methods it could implement. I've no idea how difficult this would be to implement though.

  • chrsig 5 days ago

    I face this a lot, and I think you have one very valid approach to it that I often employ. Especially if you have some centralized point to make a list of structs, or walk a tree to collect them.

    I think another other route is to provide lots of tooling to inspect the object graph -- include a pretty print function in the interface or something to that effect.

    There's equivalency between the two, the struct with pointers to implementation functions can implement an interface. Likewise, the implementation functions may call other implementations, so you haven't exactly solved the proxy problem, the state controlling which implementations becomes function local instead of on a struct. Unless you start to provide a struct with all of the context, at which point you've actually re-invented interfaces. I've grown wary of having too much of the details as function local like this because it hinders making more advanced tooling by hiding intermediate state from reflection.

    The interface provides the means to give other representations, and allows for embedding one within another, and gives runtime tooling to inspect underlying types. It gives more elegance and flexibility to the problem at language and standard library scale.

    One of the nice things is that it gives a sort of bidirectional refactoring path. If someone chose struct-of-pointers and it's at it's breaking point, those structs can stay as is and start to implement the interface with minimal effort. Likewise, if I need to boil down a hodgepodge of implementations into a really simple control flow, struct methods capture the struct and can be passed around as ordinary function pointers, so I can built a struct out of existing interface implementations.

    There's no real way around the complexity -- leaky abstractions are a thing. I think the author got it right: use interface upgrades and proxies sparingly. If they are the right tool for the job, then it's probably a good idea to go the extra mile to provide good documentation and instrumentation with extra attention to caller/callee expectations.

    • nickcw 5 days ago

      The main problem the method pointers solves for me is answering the question "can this object do X". The mere existence of a method you can type assert isn't enough.

      Returning a NotImplemented error isn't a great solution either as you might have to do quite a bit of work to get ready to call the method only to find it isn't implemented.

      Some bit of reflection magic which allowed objects to remove methods from their method sets at runtime would be perfect. You'd just use interface upgrades and you'd be sure that it would work.

      Also an available flag for each method would work too and might be neater than the function pointers.

      • chrsig 5 days ago

        > Some bit of reflection magic which allowed objects to remove methods from their method sets at runtime would be perfect. You'd just use interface upgrades and you'd be sure that it would work.

        Given that methods sets are...sets. I'd be interested at a language level, what it would look like to add some notion of set difference or expressing disjointedness in some way, e.g.,

            type ReadOnly retraction {
                Write([]byte) (int, error)
            }
        
            type ReadOnlyFile struct {
                ReadOnly
                *File
            }
         
        
        where `ReadOnlyFile` would have all of `*File`'s methods minus the methods defined in `ReadOnly`

        restriction probably isn't a good term, I could see it easily being misinterpreted, but I haven't all day to ponder it :)

        (edit: after publishing I realized `retract` might be more clear)

        I don't know if it would ultimately simplify the problem or not, but I agree that having some way to easily mask or hide a method subset could be quite nice.

        Edit:

        I want to add that the idiomatic thing to do now would be

            type ReadOnlyFile File
            / *proceed to implement every method you want accessible */
        
        
        maybe something like the following could be possible

            type ReadOnlyFile retracts *File {
                Write([]byte) (int, error)
            }
      • folays 4 days ago

        A similarity to your struct of fileops, would be Go to allow implementing an interface will nil methods. Like:

          func (f *file) Rename() = nil
        
        Then later test for nilness with: if f.Rename == nil, or with better readability:

          if v, ok := f.(interface {Rename()}); ok && v == nil // -> if true, then nil method
        
        We could define a Type Assertion to return "nil, ok" when all methods of the interface it is being passed to are implemented nil methods.

          type NotOftenImplemented interface {
            RareMethod()
            SuperRareMethod()
          }
        
          if v, ok := data.(NotOftenImplemented); ok && v == nil {
            // RareMethod() and SuperRareMethod() are nil
          }
        
          switch data.(type) {
            case NotOftenImplemented: // would be supposed to match ONLY if AT LEAST >= 1 of the methods of this interface is non-nil
          }
        
        Of course it would create a phenomenon where all callers of any interface methods would now fear of calling any method without first testing for its nillness.

        On another hand, there is also sometimes methods implemented with a single-line of panic("Not Yet Implemented") so...

  • memset 5 days ago

    Could you share a very small code snippet describing what you mean by bound method pointers?

    (Huge fan of rclone!)

    • jerf 5 days ago

      Something like this: https://go.dev/play/p/AVVz0qs2i1e

      This is much less convenient than using interfaces directly when that is possible, but comes with a lot more power. You have more insight into the "interface" value, you can use this to manually implement prototype-like behavior as seen in Javascript, you can deliberately make an object where one "method" is invoked on one object and one "method" is invoked on another, which is hard to read but may be just what you need in some situation, etc.

      There are a lot of ways to improvise on this tune.

      Also note most OO or OO-flavored languages can do this, Go has no special claim to it. But the tradeoffs are pretty similar in all those languages... you want to be sure this is what you need before you use it, because the design costs are significant.

PUSH_AX 5 days ago

I remember working in a team that used interfaces for a ton of things. Every single one of those things only ever had one concrete implementation, still to this day. A lot of those things were also purely for mocking in tests too.

Today I don’t use them, unless I need it retrospectively. Which I find is rare.

This is not a knock on interfaces as a language feature. Just inserting a random anecdote about pragmatism..

  • jen20 4 days ago

    Something one often sees in Go (which is in my mind an anti-pattern) is the supplier of an object also supplying an equivalent interface, as they would in Java or another nominative adoption system. In Go, the interface belongs to the consumer!

    • slekker 4 days ago

      But how do you avoid duplicating the interface in many places?

      We've tried both approaches, declaring the interface in one package and importing it where used but also declaring it directly colocated with the consumer.

      I've found the first approach worked better when we had to add or modify the interface methods

      • jen20 4 days ago

        You don’t. Each consumer of something declares what it needs via an interface, something concrete may be passed that implements it - including fakes for tests.

        Of course this doesn’t apply when using interfaces for polymorphic return types in a library, but that is a distinct use case for interfaces to address.

        Perhaps some of this is just taste, but this is the way I’ve found interfaces in Go to be most useful.

  • radicalbyte 5 days ago

    So you don't test the parts of your system which need mocks to be effectively tested? How do you avoid the maintenance nightmare that causes?

    I'm working on a Golang project where we've not yet introduced those kind of tests and, although it can be quicker to code initially, it's slower overall. The project relies heavily on other Open Source systems so the Golang code isn't even that complex.

    I can't imagine doing something hard and keeping it working over time without being to isolate dependencies to enable testing and keep the structure clear overall.

    • the_gipsy 5 days ago

      The question is, are those interfaces used for anything but mocking tests? If not, could there be a better solution?

      • valzam 5 days ago

        I keep hearing this but I don't quite understand what the alternative is? I get that maybe introducing interfaces within your own business domain CAN be overused but certainly any external depedency should be fronted by an interface so the rest of the code can be tested without spinning up test http servers, database containers etc. all over the place. It just feels like a really lightweight, simnple thing to have an interface for some API that you are calling. It also means you are less depedent on the concrete implementation of the API and don't leak types and behaviour from systems you don't control into your codebase

        • dgb23 5 days ago

          Instead of pushing the external, side-effecting thing down the stack, I put it up the stack and call the logic that generates the data for it from there. Now I don't need to mock an interface, I just need to test data in/out.

          If you do this consistently, then you have much of the side effecting code residing next to each other per process/input handler etc.

          It's not always feasible or the right thing. But it's a good default way of structuring code and makes testing much more straight forward. And it makes more clear "at a glance" what happens because you can reason more locally about the side effecting parts.

          > It also means you are less depedent on the concrete implementation of the API and don't leak types and behaviour from systems you don't control into your codebase

          You are always dependent on that. Whether you hide it down the stack or lift it up, you still need to put your data into a specific shape, do the same essential checks etc.

          • randomdata 4 days ago

            > Instead of pushing the external, side-effecting thing down the stack, I put it up the stack and call the logic that generates the data for it from there.

            I've never seen Go code that does anything else. That's not to say it doesn't exist, but it's seemingly not the norm amongst Go developers (probably exactly for the reasons you describe). That doesn't address what the parent is talking about, though. You have the exact same problem spoken of no matter where in the stack you put it.

        • the_gipsy 8 hours ago

          Could be structural duck typing, like typescript does. Or just include some other monkey patch mechanism specific for tests.

        • afavour 5 days ago

          This is a situation where I enjoy coding in TypeScript. You don’t need to front with interfaces, you just mock objects that map 1:1 with the expected API. So you write your code targeting the standard Fetch API then just pass in a mock when testing.

        • ashishmax31 5 days ago

          +1 to this. Especially with go, most of the (non-integration) tests I end up writing are behaviour driven tests. I'm not really sure how one can write these sort of tests without interfaces and mocks.

          • randomdata 5 days ago

            One could use build tags to provide alternative implementations over a concrete type, I suppose.

            But like with everything in life, there is no right solution. Just different tradeoffs. If interfaces best align with the trades you are willing to make, go for it.

        • wavemode 5 days ago

          To be honest, if I were writing software in Go I would just test it end-to-end.

          If the software's logic is too complex and fiddly to lend itself to straightforward end-to-end testing (i.e. enterprise software), I just wouldn't write it in Go. I'd choose a higher level language like Python or Java where mocking (and other kinds of dynamism) are straightforward and require no boilerplate.

    • PUSH_AX 5 days ago

      I’m saying we did test parts that required mocks, and we used interfaces as a vehicle to streamline that. Rightly or wrongly.

      • PUSH_AX 5 days ago

        I’ve just realised you understood and we’re asking present tense I think. The truth is I also do less unit testing, identifying instead where I know confidence is needed and where it’s not.

karmakaze 5 days ago

This is basically a hack to get pragmatic performance at a cost of comprehensibility. It would have been better if Go allowed union types then a func could be declared to use interface A|B efficiently. Passing a narrow interface when a wider implemented one could get used is lying about the actual "interface" in the signature. Added to that is that Go's interfaces are structural (which I think are great), but could lead to accidental misuse by passing in interface A for an object that also has method X for unrelated purposes that co-incidentally satisfies interface B which the called func on A magically uses.

> Like all articles about Go’s interfaces, we are obligated to start with Go’s io package.

Also the io package, or stdlib in general is not a good place to look for good patterns to use in Go. Numerous antipatterns are used in the name of performance. The principles for stdlib authors and recommendation for Go developers are different. As an example io functions can return a value AND an error--and whether to continue or not depends on the specific error (as some are benign). It's better that I don't name an example as you should always be on the lookout for (until having learned) them.

  • 38 5 days ago

    > It's better that I don't name an example as you should always be on the lookout for (until having learned) them.

    "this gun will shoot you in the foot in some cases. It's better that I don't name an example as you should always be on the lookout for (until having shot yourself in the foot) them"

    • karmakaze 4 days ago

      They are all documented in the io package. There's no point in telling you just one of them. But you have to RTM which was my point.

rollulus 5 days ago

Note that this is not necessarily a great thing to overuse. Also note that the article is 10 years old.

Relying on such upgrades sort of introduces a dark and fuzzy part of the API. Go’s http pkg is a notorious one, with how a http.ResponseWriter can also be a Flusher and Hijacker and I don’t know what else. If you in your middleware want to wrap it, you need to implement those interfaces as well, or the functionality is lost. But they’re not part of the “visible” API, they’re type assertions buried deep in the standard library, good luck with that. For this reason Go 1.20 introduced the http.ResponseController.

  • never_inline 5 days ago

    I encountered this and went through some corners of the standard library.

    shameless plug to my own blog post: https://mahesh-hegde.github.io/posts/go-interface-smuggling/

    • quectophoton 4 days ago

      Wait what.

      For functionality like that[1] I would have expected them to fallback to `bufio.Peek` if the `fs.File` is not an `io.Seeker`. Sure, it's a bit slower, but it would just work. If someone implements custom `fs.File`, then it's up to them implementing any additional interface that allows better performance.

      It's the same idea as `io.Copy` trying to use first `io.WriterTo` or `io.ReaderFrom` if implemented, and then falling back to a manual copy.

      The thought process should be: First try to ask for an interface that has all methods you require (that's like the whole point of an interface). If you can't do that, then try to work with the methods you have, and then mention that performance can be improved if additional interfaces are implemented. Only if when you can't do that, you fail with a runtime error complaining about missing methods you didn't ask for in the interface signature.

      I might be missing some reason that prevents them from using `bufio.Peek` though.

      [1]: Link to the use of Seek, for reference: https://github.com/golang/go/lob/e8ee1dc4f9e2632ba1018610d1a...

    • rollulus 5 days ago

      If you never encountered this you never created your own ResponseWriter, or you did it and inadvertently broke its Flushing, Pushing and Hijacking capabilities. It’s not more complicated than this.

geoka9 4 days ago

> While it just so happens that all the ResponseWriters that net/http give you implement (e.g.) CloseNotifier, there’s not really any way for you to know that without reading the source.

Go editor tooling helps with this: gopls can do this nowadays (e.g. `lsp-find-implementation` in Emacs) and go oracle/guru may have already supported this back in 2014. It works both for finding implementations of an interface and finding interfaces implemented by a type.

McMini 5 days ago

While the article highlights interface upgrades, it overlooks potential downsides like increased complexity in debugging. Anyone experienced issues with this?

MrBuddyCasino 5 days ago

Am I missing something or is this a lot of words to describe the equivalent of the JVMs instanceof / type casting operator?

  • neonsunset 5 days ago

    It is. What's with the languages and their communities trying to reinvent worse wheel and label it with a new word all the time?

        if (thing is IFasterThing ft)
        {
            ft.FastPath(arg1, arg2);
        }
    • blev 5 days ago

      The author’s mistake was assuming that interfaces are unique to Go. I think it’s still a helpful article as someone who works frequently with interfaces because, while I “discovered” the interface upgrade idea on my own, I had never considered the proxy problem.

      • philosopher1234 5 days ago

        Go’s interfaces are unique in that they are structural, which makes this problem more difficult to solve.

yawz 5 days ago

The title needs "2014" to be added.

  • dang 5 days ago

    Added. Thanks!

hmage 5 days ago

November 5, 2014

  • dang 5 days ago

    Year added above. Thanks!