java.nio DirectoryStream vs java.io.File#list or File#listFiles performance and memory usages

[shashank dwivedi]

I want to collect directory structure to a given depth (It can be huge recursive structure!).
Which approach should perform better and consumes less memory?

Approach 1: Using DirectoryStream

    public void collectFilesTillDepth(Path path, int depth) {
        if(depth == 0){
            IPath p = IPath.TYPE.LEVEL.equals(rootPath.getType()) ? new LevelPath(path, rootPath.getRT()) : new BitFilePath(path, rootPath.getRT());
            p.accept(flushVisitor);
            return;
        }

        try (DirectoryStream<Path> ds =
                     Files.newDirectoryStream(path)) {
            for (Path p : ds) {
                collectFilesTillDepth(p, --depth);
                depth++;
            }

        } catch (IOException e) {
            e.printStackTrace();
        }
    }

OR

Approach 2:

// Not tested and just for getting opinions
    public void collectFilesTillDepthIO(File file, int depth) {
        if(depth == 0){
            System.out.println(file.getPath());
        }
        File[] children = file.listFiles();
        if(null != children && children.length > 0)
        for (File child : children) {
            collectFilesTillDepthIO(child, --depth);
            depth++;
        }
    }

Q. As java doc says -

    * <p> Note that the {@link java.nio.file.Files} class defines the {@link
    * java.nio.file.Files#newDirectoryStream(Path) newDirectoryStream} method
    * to open a directory and iterate over the names of the files in the
    * directory. This may use less resources when working with very large
    * directories.

I am visiting directories in DFS manner(recursive approach), will java.io.File#listFiles perform better than DirectoryStream as stream entries won't be garbage collected till whole structure is visited (not sure on this) causing more memory usages?

[Stephan van Hulst]

Why are you worried about this? Do you have reason to believe whatever approach you're using is not sufficient?

Use the right tool for the right job, and then test if performance is acceptable, before wasting your time on trying to optimize parts that are fine as they are.

In this case, it looks like you want to walk a tree of paths. You even wrote a recursive function to do it. Instead, use Files.walk() or Files.walkFileTree().

[shashank dwivedi]

I wanted to list files at given depth only.

[shashank dwivedi]

I am sorry. FileWalkTree already gives what I wanted. Just was a little bit concerned about memory usages as it also uses DirectoryStream.

[Stephan van Hulst]

First test, then worry.

[Rob Spoor]

shashank dwivedi wrote:

        if(null != children && children.length > 0)

That line alone should be enough reason not to use File.list or File.listFiles. Instead of throwing an exception, they return null. You have no idea why though. And it also doesn't return an empty array so you have to add the null check. When using Files.walk, Files.walkFileTree, Files.list or Files.newDirectoryStream, you get a proper exception that tells you why the files can't be listed / traversed.