• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

How does one debug a stream pipeline?

 
Ranch Foreman
Posts: 587
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I wanted to learn what was happening in this program by Roberts/Zaikin.
Using IntelliJ and running this stream pipeline in the debugger,
I was finding that it is not just stepping through each statement and observing the values in the watch window.
Even trying to put a peek(System.out::println) is not possible in the intermediate steps, because the generated objects do not support it.
It was much simpler using a non-stream program
See screenshot below that shows none of the intermediates can be seen in the watch window.

javaranch0.png
[Thumbnail for javaranch0.png]
 
Bartender
Posts: 3909
43
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Anil:

I am glad you use our sample quiz for Java learning.

If you are using ItelliJ Idea (even free Community edition) you can use advanced debugger feature to check how pipeline processes data.

Hope screenshots explain it.
pipeline_debug_1.jpg
[Thumbnail for pipeline_debug_1.jpg]
pipeline_debug_2.jpg
[Thumbnail for pipeline_debug_2.jpg]
 
Anil Philip
Ranch Foreman
Posts: 587
1
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Mikalai Zaikin wrote:
I am glad you use our sample quiz for Java learning.

If you are using IntelliJ Idea (even free Community edition) you can use advanced debugger feature to check how pipeline processes data.

Hope screenshots explain it.



Your quizzes should be on the list of recommended resources for certification - I should have done them earlier.
(Suggestion - it would help if the quizzes were organized in some way so we can keep track of the ones we have done and our progress.
Currently they are not even numbered.)

Wow, I did not know the pipeline could be examined like this!
I am using the free Community Edition, but I get "internal error". see screenshot.
I had to click a different widget than in your screenshot to get this trace feature.
It is not working for me.
javaranch0.png
[Thumbnail for javaranch0.png]
 
Anil Philip
Ranch Foreman
Posts: 587
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Seems like an open-closed issue that is seen often in different versions. I opened a ticket here https://youtrack.jetbrains.com/issue/IDEA-346690/Stream-trace-has-internal-error-on-IntelliJ-IDEA-2023.3.4-Community-Edition
 
Mikalai Zaikin
Bartender
Posts: 3909
43
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Can you put a breakpoint here:
pipeline_debug_3.jpg
[Thumbnail for pipeline_debug_3.jpg]
 
Anil Philip
Ranch Foreman
Posts: 587
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I did and same result. Also in Community Edition I had to press a different widget to get Stream Trace.
screenshot attached.
javaranch0.png
[Thumbnail for javaranch0.png]
javaranch1.png
[Thumbnail for javaranch1.png]
 
Anil Philip
Ranch Foreman
Posts: 587
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
In idea.log I see this exception

2024-02-19 21:20:30,661 [ 196382]   INFO - #c.i.c.s.BuildManager - BUILDER_PROCESS [stderr]: Be careful, logger will be shut down earlier than application: Unable to make field private static java.util.IdentityHashMap java.lang.ApplicationShutdownHooks.hooks accessible: module java.base does not "opens java.lang" to unnamed module @2f4d3709
2024-02-19 21:20:30,690 [ 196411]   INFO - #c.i.c.s.BuildManager - BUILDER_PROCESS [stdout]: Build process started. Classpath: C:/Program Files/JetBrains/IntelliJ IDEA Community Edition 2022.2.3/plugins/java/lib/jps-launcher.jar
2024-02-19 21:20:30,707 [ 196428]   INFO - #c.i.a.d.c.d.t.TimeSpanUserActivityDatabaseThrottler - Starting activity runconfig.running with id 2064515432
2024-02-19 21:20:37,835 [ 203556]   WARN - org.apache.velocity.deprecation - configuration key 'resource.loader' has been deprecated in favor of 'resource.loaders'
2024-02-19 21:20:37,837 [ 203558]   WARN - org.apache.velocity.deprecation - configuration key 'includes.resource.loader.class' has been deprecated in favor of 'resource.loader.includes.class'
2024-02-19 21:20:38,365 [ 204086]   INFO - #c.i.r.e.ExtractLightMethodObjectHandler - Use reflection to evaluate inaccessible members
2024-02-19 21:20:38,838 [ 204559]   INFO - #c.i.r.e.ExtractGeneratedClassUtil - Replace constructor: GeneratedEvaluationClass
2024-02-19 21:20:38,839 [ 204560]   INFO - #c.i.r.e.ExtractGeneratedClassUtil - Replace method: invoke
2024-02-19 21:20:38,839 [ 204560]   INFO - #c.i.r.e.ExtractGeneratedClassUtil - Replace method: myRes
2024-02-19 21:20:38,840 [ 204561]   INFO - #c.i.r.e.ExtractGeneratedClassUtil - Replace method: result
2024-02-19 21:20:41,319 [ 207040] SEVERE - #c.i.d.i.InvokeThread - Internal error
com.intellij.debugger.streams.diagnostic.ex.TraceCompilationException: Internal error
at com.intellij.debugger.streams.action.TraceStreamAction$1.compilationFailed(TraceStreamAction.java:133)
at com.intellij.debugger.streams.trace.EvaluateExpressionTracer$1.errorOccurred(EvaluateExpressionTracer.java:86)
at com.intellij.debugger.engine.JavaDebuggerEvaluator$1.threadAction(JavaDebuggerEvaluator.java:96)
at com.intellij.debugger.engine.events.DebuggerContextCommandImpl.contextAction(DebuggerContextCommandImpl.java:72)
at com.intellij.debugger.engine.events.SuspendContextCommandImpl.action(SuspendContextCommandImpl.java:48)
at com.intellij.debugger.engine.events.DebuggerCommandImpl.run(DebuggerCommandImpl.java:40)
at com.intellij.debugger.engine.DebuggerManagerThreadImpl.processEvent(DebuggerManagerThreadImpl.java:155)
at com.intellij.debugger.engine.DebuggerManagerThreadImpl.processEvent(DebuggerManagerThreadImpl.java:27)
at com.intellij.debugger.impl.InvokeThread.lambda$run$0(InvokeThread.java:140)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$1(CoreProgressManager.java:192)
at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$12(CoreProgressManager.java:610)
at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:685)
at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:641)
at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:609)
at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:78)
at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:179)
at com.intellij.debugger.impl.InvokeThread.lambda$run$1(InvokeThread.java:126)
at com.intellij.openapi.project.DumbService.runWithAlternativeResolveEnabled(DumbService.kt:331)
at com.intellij.debugger.impl.InvokeThread.run(InvokeThread.java:126)
at com.intellij.debugger.impl.InvokeThread$WorkerThreadRequest.lambda$run$0(InvokeThread.java:49)
at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:218)
at com.intellij.debugger.impl.InvokeThread$WorkerThreadRequest.run(InvokeThread.java:48)
at com.intellij.openapi.application.impl.ApplicationImpl$2.run(ApplicationImpl.java:249)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at com.intellij.util.concurrency.ContextCallable.call(ContextCallable.java:32)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.intellij.util.concurrency.ContextRunnable.run(ContextRunnable.java:27)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:702)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:699)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:699)
at java.base/java.lang.Thread.run(Thread.java:840)

 
Saloon Keeper
Posts: 27819
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Seems like peek should be able to do it, if one knew the right syntax. Something like peek(e -> log.debug(e) ) give or take, I'd imagine.

Last time I had to debug a stream, I did it via brute force, and that's about as much fun as debugging the old-time YACC code. Too much of the same thing to set decent breakpoints and too much hidden stuff. I did find/fix eventually though.
 
Anil Philip
Ranch Foreman
Posts: 587
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:Seems like peek should be able to do it,


peek() is not supported on all data structures. For instance on maps, that are in this pipeline, peek will not work.
It is actually dangerous in my humble opinion, to have complex pipelines because bugs will never get detected nor fixed - because the code is hard to understand.
 
Marshal
Posts: 4519
572
VSCode Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You should be able to peek into a Map and Map.Entry.  I'm running Java 21; maybe it is different in other versions.

 
Anil Philip
Ranch Foreman
Posts: 587
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Ron McLeod wrote:You should be able to peek into a Map and Map.Entry.


Are you able to put a peek into the different points over here?
From the original example, I wasn't able to.
 
Ron McLeod
Marshal
Posts: 4519
572
VSCode Eclipse IDE TypeScript Redhat MicroProfile Quarkus Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If you mean inside an operation, then no.
 
Master Rancher
Posts: 4854
74
  • Likes 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Anil Philip wrote:peek() is not supported on all data structures. For instance on maps, that are in this pipeline, peek will not work.


It's not just a binary yes/no based on the data structure - it usually has more to do with the combination of operations in the stream, possibly in combination with the data structure.  As the peek() API says:

In cases where the stream implementation is able to optimize away the production of some or all the elements (such as with short-circuiting operations like findFirst, or in the example described in count()), the action will not be invoked for those elements.


I sometimes use a stream.map(e -> { doSomething(e); return e; }) instead of a peek(e -> doSomething(e)).  It's a bit less likely to be optimized away.  Though in some cases, it still can be.  E.g. if the terminal operation is count(), you may not need to evaluate a map() just before that, because the mapping can't possibly affect the count.

Anil Philip wrote:It is actually dangerous in my humble opinion, to have complex pipelines because bugs will never get detected nor fixed - because the code is hard to understand.


Well, it's true that streams can be harder to debug.  And Collectors, and the groupingBy() overloads in particular, are among the most complicated things you can do with a stream.  I find them very useful - usually, more than half of the work for me is getting it to compile in the first place, meaning to get all the types to line up correctly.  Once I've done that, often the result is simply right, with no further work needed on my part.  Or maybe I need to study the result to see how it compares with what I wanted, and modify accordingly.  But, after some time getting used to streams, I now find it's usually quicker and easier for me to use them, compared to coding the old fashioned way. Not always, but more often than not.

It is true, however, that debugging can get more difficult.  Especially with Collectors, which as you say, do not really provide you with a place to put a logging statement or breakpoint.

Unless, of course, you make such places.

Here are some utility functions which I have found useful:


Note that inside the last private log() method you can use whatever output format and/or logging framework you want.  I use System.out here because everyone has it and it's easy to understand - but feel free to use a better logging framework.

Anyway, once you have that, you can use it to decorate the various collectors you have.  I recommend extracting them to separate variables on separate lines for readability (e.g. using IntelliJ's Refactor -> Introduce variable).  For the code

I extracted those nested collectors, thus:

and then applied those utility methods I just showed you, to obtain:

     
Note that the indentLevel stuff is intended to mirror the nested nature of the collectors - logs for the outside the collectors are at indent level 0, the outermost collector logs at indent level 1, the next inside that is at level 2, etc.  The declarations necessarily start with the innermost and work outward, from 3 to 2 to 1, which looks a little backwards, but it works.  If getting the indent levels worked out is too much trouble, you can remove that argument.  But personally, I find it useful.

Last, I also added to toString() to the JavaExam class:

And I get the following output:

That may seem like a lot of work... but, those utility methods can be reused in other projects, maybe tweaking them depending on your use case.
 
Tim Holloway
Saloon Keeper
Posts: 27819
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Very impressive! Have you considered packaging this as a testing library that could maybe go into the world Maven archives?
 
Mike Simmons
Master Rancher
Posts: 4854
74
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks!  I hadn't really considered it; in fact I just came up with most of it.  But perhaps it's worth doing something like that...
 
Anil Philip
Ranch Foreman
Posts: 587
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Mike Simmons wrote:Thanks!  I hadn't really considered it; in fact I just came up with most of it.  But perhaps it's worth doing something like that...


Honestly, I did not understand - it all went over my head. I am not very intelligent.
But I am convinced this is a shortcoming of Java.
I am thinking of playing with the Streams API, modifying the implementing code just to experiment and try out an idea.
Is it very hard to understand how-to, and modify and build the JDK on my local machine? Has anyone outside of Oracle done it?
 
Ranch Hand
Posts: 49
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Anil Philip wrote:

Mike Simmons wrote:Thanks!  I hadn't really considered it; in fact I just came up with most of it.  But perhaps it's worth doing something like that...


Honestly, I did not understand - it all went over my head. I am not very intelligent.



Don't say that. You just haven't learned it yet. I often find myself overwhelmed by how much their is to learn and how complex some subjects seem, but that's not because I'm not smart. You're plenty intelligent, give yourself time to learn
 
Tim Holloway
Saloon Keeper
Posts: 27819
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Anil Philip wrote:

Mike Simmons wrote:Thanks!  I hadn't really considered it; in fact I just came up with most of it.  But perhaps it's worth doing something like that...


Honestly, I did not understand - it all went over my head. I am not very intelligent.
But I am convinced this is a shortcoming of Java.
I am thinking of playing with the Streams API, modifying the implementing code just to experiment and try out an idea.
Is it very hard to understand how-to, and modify and build the JDK on my local machine? Has anyone outside of Oracle done it?



Lots of people have done it, but remember that there's more than one JDK. Oracle has one, Open Source has at least one (OpenJDK), IBM has one, and so forth.

I wouldn't recommend building your own JDK just to understand streams. It wouldn't help much anyway because the gnarly thing about debugging streams is that the same code gets called over and over until your eyes glaze.

You can't easily obtain the source for Oracle or IBM J9, but anyone can "git clone" the OpenJDK source and build it. Like many open-source projects, it's just a matter of cloning or downloading/unzipping, running the "configure" to get it to detect your OS environment and build makefiles, then "make" to build it. Mostly it just takes a while to do all that compiling and such, since it's a large system.

 
Anil Philip
Ranch Foreman
Posts: 587
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:
I wouldn't recommend building your own JDK just to understand streams. ...
but anyone can "git clone" the OpenJDK source and build it. Like many open-source projects, it's just a matter of cloning or downloading/unzipping, running the "configure" to get it to detect your OS environment and build makefiles, then "make" to build it. Mostly it just takes a while to do all that compiling and such, since it's a large system.



Not to understand streams, but to try out something small.
I was wondering if one needs to do this on a large computer with a terabyte or more.
I have 200Gb free space on my laptop SSD.

 
Tim Holloway
Saloon Keeper
Posts: 27819
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The build instructions in the source archive are pretty detailed and they say you can build in 6GB of disk space, but it's disk-intensive so the faster the disk, the better, e.g., SSD.

For CPU/RAM:

Build instructions wrote:
At a minimum, a machine with 2-4 cores is advisable, as well as 2-4 GB of RAM. (The more cores to use, the more memory you need.) At least 6 GB of free disk space is required.



Have fun, and tell us how it went!
 
Anil Philip
Ranch Foreman
Posts: 587
1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:The build instructions in the source archive are pretty detailed and they say you can build in 6GB of disk space, but it's disk-intensive so the faster the disk, the better, e.g., SSD.
Have fun, and tell us how it went!


Thank you - will do!
 
Marshal
Posts: 8869
637
Mac OS X VI Editor BSD Java
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Cowgratulations, Anil Philip! Your topic has been published in our CodeRanch's February 2024 Edition Journal
 
Yes, my master! Here is the tiny ad you asked for:
a bit of art, as a gift, that will fit in a stocking
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic