Generics solve when passing parameter but not returning a value

[Paul Houle]

I've got a situation that has me scratching my head.  I am hacking on something that is a simplified version of the JDK 8 Streams framework that mixes in some ideas from Python and ML.  Mostly I am trying to learn more about writing internal DSLs in Java.

So I like writing

https://github.com/paulhoule/pidove/blob/c3cc0289ea4ee6424c711f977a7fe777613dad05/src/test/java/com/ontology2/pidove/checked/TestCollect.java#L57

var grouped = collect(groupingBy(i->i % 3, toSet().andThen(Set::size)), that);

in particular

toSet().andThen(Set::size)

which the definition for toSet() is

https://github.com/paulhoule/pidove/blob/c3cc0289ea4ee6424c711f977a7fe777613dad05/src/main/java/com/ontology2/pidove/checked/Collectors.java#L67

    public static <X> SimpleCollector<X, Set<X>> toSet() {
        return toCollection(HashSet::new);
    }

and the definition of andThen() is

https://github.com/paulhoule/pidove/blob/c3cc0289ea4ee6424c711f977a7fe777613dad05/src/main/java/com/ontology2/pidove/checked/SimpleCollector.java#L11

public record SimpleCollector<Input, Result> (Supplier<Result> supplier,BiConsumer<Input, Result> accumulator) implements AnyCollector<Input, Result> {
    public <FinalResult> Collector<Input,Result,FinalResult> andThen(Function<Result,FinalResult> finisher) {
        return Collector.of(supplier(), accumulator(), finisher);
    }
}

So far so good,  but if I want to create a prepackaged "distinct count" collector it looks like:

https://github.com/paulhoule/pidove/blob/c3cc0289ea4ee6424c711f977a7fe777613dad05/src/main/java/com/ontology2/pidove/checked/Collectors.java#L57

    public static <X> Collector<X,Set<X>,Integer> countDistinct() {
        return Collectors.<X>toSet().andThen(Set::size);
    }

it works if I write it like that,  but I have to include the type variable explicitly with

Collectors.<X>

for it to compile,  otherwise I get this error

incompatible types: no instance(s) of type variable(s) FinalResult exist so that com.ontology2.pidove.checked.Collector<java.lang.Object,java.util.Set<java.lang.Object>,FinalResult> conforms to com.ontology2.pidove.checked.Collector<X,java.util.Set<X>,java.lang.Integer>

I'd much rather not have to specify that type variable and it seems I shouldn't have to.  Particularly I look at how Function<X,Y>.andThen() works.  So I'm wondering,  am I doing some little thing wrong here or am I in some deep trouble with type erasure?

It is on Github at

https://github.com/paulhoule/pidove/

and there is a tag q1 for this exact code.  So if you want to fire up your IDE or do maven install it is easy,  the one thing is you need JDK 17 to build it.

Thanks!

[Paul Clapham]

This  is mostly way over my head, but one thing I notice is that Collectors.toSet() is a static method whereas Function.andThen() is an instance method. That suggests to me that maybe the compiler uses the type of the Function object to help with type inference, whereas there's no Collectors object to help with that. Does that sound plausible?

By the way, welcome to the Ranch!

[Stephan van Hulst]

Type inference in the backwards direction usually doesn't extend beyond chained method calls.

The problem is that while the return type of your countDistinct() method can be used to infer the type argument of the andThen() method call, the andThen() method call is not used to infer the type argument for the toSet() method call.

That means that if you don't explicitly specify the type argument for the toSet() call, it defaults to Object, which is not compatible with whatever was inferred for andThen().

There's no real way around this. If you don't like the syntax for type witnesses, consider breaking up your statement in two seperate stements like this:

SimpleCollector<X, Set<X>> toSet = toSet();
return toSet.andThen(Set::size);

Note that clients of your API will run into the same issues if they want to chain collectors together. To make life easier for them, consider adding overloads with a method parameter of type Class if the original method didn't have any parameters at all (such as the toSet() method). This won't help them if they're writing generic methods, but it might help if they're working with concrete types:

toSet(String.class).andThen(Set::size);

This is not in general recommended though. You're adding extra processing overhead just to avoid using a certain syntax that's already provided by the language.

For this specific case it might be possible to sidestep the whole issue completely if you use type bounds correctly. I don't have a compiler here to check if it works, but see what happens if you change your method signatures to the following:

public <RR> Collector<? super T, ?, RR> andThen(Function<? super R, ? extends RR> finisher)

public static Collector<? super T, ?, Integer> countDistinct()

[Paul Houle]

Stephan,  that's a great answer.

Here is what I tried.  Changing the signature of countDistinct() did the trick

    public static <X> Collector<? super X, ?, Integer>countDistinct() {
        return toSet().andThen(Set::size);
    }

that compiles just fine,  with the compiler figuring out the type of toSet(); after this I changed the signature of andThen() and it still compiles.

I wound up,  however,  giving up on writing my own Collectors because I found (1) there was nothing really wrong in my mind with the collectors from the Stream package and (2) there is a lot of quality engineering in the java.util.stream Collectors worth keeping.  So I am just using the java.util.stream Collectors with my own methods that work on Iterables.

It does sober me a little about the API I can get away with,  I notice the stream Collectors use a static method to do the work of andThen

static <T, A, R, RR> Collector<T,A,RR> collectingAndThen(Collector<T,A,R> downstream, Function<R,RR> finisher)

for which the types resolve in more situations.  On one hand I like Feynmanesque solutions that look a lot simpler than they really are,  I also don't want to make users of the API fight with the generics system.  It does push me in the "put the verb on the left" style of static methods as opposed to the "middle/right verb" style of chained calls.

Is there a good resource (book, problem sets, etc.) for really mastering generic methods in Java?

[Stephan van Hulst]

I prefer chained method calls over static calls as well (I believe this is called "fluent API"), but in some unfortunate cases it just doesn't make our life easier.

Most of my experience with generics comes from lots of practice with the streams API. It's well worth carefully inspecting the type parameters used in the standard library, and figuring out why the type bounds are given the way they are. I also learned a lot from writing custom implementations of the Collector interface.

This week I don't really have the opportunity to look at your Github repository, but if it contains any stream-like operations that were adapted for Iterable, note that you can relatively easily convert Iterable to Stream using the StreamSupport and Spliterators classes.

[Campbell Ritchie]

Stephan van Hulst wrote:. . . Most of my experience with generics comes from lots of practice with the streams API. . . .

Maybe that is because it is the Streams API is the only commonly used API to use complicated generics.

[Paul Houle]

My favorite internal DSL in java is

https://www.jooq.org/

because it lets you write code in a real programming language (SQL stored procedures) as an internal DSL, see...

https://www.jooq.org/javadoc/latest/org.jooq/org/jooq/Block.html

I'm pretty convinced it is possible to write Java-in-Java and I started hacking on this a year ago

https://github.com/paulhoule/ferocity/blob/main/ferocity0/src/test/java/com/ontology2/ferocity/UrFierce.java

the idea of ferocity is that you can write Java syntax trees in a style kinda like JooQ or Lisp (like in the above unit tests) and then either evaluate the syntax tree with a primitive interpreter or turn it into Java code that can be fed back through Javac.

It sounds crazy but I think it is a powerful strategy for code generation,  the plan I had was to write the generation 0 ferocity,  get it up to the point where it could compile stubs for the Java standard library so if you want to write say

ClassLoader c = ...;
c.findResource("something");

you would write

import 𝔣.java.lang.Classloader.findResource;
...
Expression<ClassLoader> c = ...;
findResource(c,of("something"))

where of generates the literal (an Expression),  instead of passing objects in you pass in Expressions.  That 𝔣 is a Unicode character and these are used all over the place,  for
instance sometimes it adds double-angle brackets to fill in the types of a function name if it is ambiguous because type erasure means you can't distinguish between Expression<This> and Expression<That>.

What I found out was that language that the expression interpreter can process wants to be bigger than Java,  for instance you can quote an Expression and have it be processed as an Expression in the interpreter to do 'syntactic macros' like they have in LISP.  You can't (I think) serialize those to Java code but you can interpret them.  Similarly there is a concept that includes variables, parameters that it is useful to define new kinds of.

I'd convinced myself that there wasn't any fundamental problem,  just a lot of work,  to realize this and figured the easy way to do it was write ferocity1 using ferocity0 to do code generation but then I got distracted by some Arduino project.  I probably should have finished it before starting pidove but...

[Campbell Ritchie]

Please use the code button (more details here) rather than writing the code tags by hand; I have corrected whatever you got wrong, and doesn't it look better I think you wrote Java rather than java.
Is the 𝔣 in line 1 a standard character on your keyboard? How would you get it otherwise?

[Paul Houle]

I usually cut and paste unusual characters.  Pages like

https://en.wikipedia.org/wiki/Fraktur#Unicode

are easy to find.  If you were using this library to write something in an IDE you might never need to type or paste the 𝔣,  except maybe in the POM if it isn't the default,  because classes are code generated for that and put on the classpath.

Usually I'd type the name of a method I want into IntelliJ Idea and it would have a red squiggle then I could pick the namespace and it writes the import.  The IDE & Language has so many ways to find things and save you from writing "the.namespace.of.my.product" over and over again that unusual characters in Java code are not a burden.

I like how unusual characters can create a feeling that you're changing the language like FORTH or LISP by taking advantage of what you can in the language.  For instance that 𝔣-prefix acts like a parallel world to ordinary Java classes.  For instance you

import 𝔣.java.lang.String.getBytes

to get a static function which corresponds to

String.getBytes()

in the real world.  I think it is fun but you could change it to something respectable like "com.example.fierce" in the POM file.

[Campbell Ritchie]

Thank you

I haven't thought of String#getBytes() as being static.

[Paul Houle]

Every class that the generator targets gets fierce methods that mirror the real methods,  it is like

Expression<String> some_string = of("Some String")
Expression<byte[]> expression = getBytes(some_string);

and you've created a Java expression tree that can be serialized as

getBytes("Some String")

if you do

expression.asSource()

or gives you

{83, 111, 109, 101, 32, 83, 116, 114, 105, 110, 103}

if you do

expression.evaluate()

The ferocity0-maven module is a maven plugin does code generation using that representation for expressions and also methods and classes,  when the ferocity-stdlib module runs that generator to build the stubs we are talking about.

Why would somebody do this?

Because the Java language has the 8 primitive types you find the generics system isn't really complete.  For instance the Streams API has IntStream, DoubleStream, etc. to reduce boxing.  Writing pidove in ordinary Java I am not going to make an IntIterable, DoubleIterable, etc. because it is pretty tedious.

With  ferocity, pidove could have a template set up such that you can write something like

generatePrimitiveIterable("IntIterable", int.class);

and it substitutes "int" into the right places into an IntIterable class,  like the template specialization that might come with Valhalla.  Kinda the best thing I've figured out to write with ferocity is ferocity itself,  which is admittedly self-referential,  but I think there is enough complexity with primitive types and other details that I'd rather write the simplest possible system for ferocity0 (it supports expressions but not statements,  doesn't support if, for,  things like that) and then write ferocity1 (which does all of java) using ferocity0 as a code generator.

It's the kind of thing LISPers do all the time.