Get folder size/ACL from millions folders

[igor88]

hello

googled some on stackoverflow...etc - but too old info

sometimes i need to get folder size/acl for millions of folders

powershell is working, but its too slow - and it uses a lot of ram even without putting all object to ram

so - what .net methods/classes should i use in 2023?

 

[Skydiver]

If you make full use of pipelines in PowerShell it should not be a memory hog unless you are doing sorting or grouping. With sorting or grouping, it has no choice but to retain things in memory to perform that operation. Unfortunately PowerShell prefers to run single threaded and you need to bend over backwards to get some parallelism working.

If you really must use C#, use the EnumerateFiles() method to get the files one a time so that you get each filename one at a time instead of as an array. (This is what newer versions of PowerShell also use.) Get the file size and ACL information from that file and be extreme paranoid about disposing and disposable objects instead of relying on the garbage collector. Write the filename, file size, and ACL information out into a file or database so that you don't keep a growing list on memory. Partition your scans so you can do them in parallel instead of sequentially.

 

[igor88]

If you make full use of pipelines in PowerShell it should not be a memory hog unless you are doing sorting or grouping. With sorting or grouping, it has no choice but to retain things in memory to perform that operation. Unfortunately PowerShell prefers to run single threaded and you need to bend over backwards to get some parallelism working.

If you really must use C#, use the EnumerateFiles() method to get the files one a time so that you get each filename one at a time instead of as an array. (This is what newer versions of PowerShell also use.) Get the file size and ACL information from that file and be extreme paranoid about disposing and disposable objects instead of relying on the garbage collector. Write the filename, file size, and ACL information out into a file or database so that you don't keep a growing list on memory. Partition your scans so you can do them in parallel instead of sequentially.

thanks. but it need folder info, not files

RAM usage in powershell found - debug info _)

 

[Skydiver]

There are also EnumerateDirectories() methods.