Speed up search using filters

Help & Support
1

Hello,

based on this FAQ I often refer to:

I'm trying to look for all files in a specific folder without losing time to search through all unnecessary paths.

Basically, what I'm looking for is a list of all files located in :
\Serveur\c$\Users*\AppData\Local\Microsoft\Terminal Server Client\Cache*

If I put this in the Full Path filter, it works but it is slow and I can see it goes through every folders in the hierarchy.
I have tried to add Subfolder Location filter based on "*\AppData\Local\Microsoft\Terminal Server Client\Cache*" , but I didn't manage to find the correct parameter.
I added extra *\ in font of the path, but it doesn't work.

Would you know how to make it work ?

Thank you

2

Where are you starting the search from?

If a folder doesn't match that subfolder condition, it and everything below it will be excluded from the search.

If you're starting it at the root of a drive, nothing will match. You won't reach the AppData folders because their parents have already been excluded.

Even the AppData folders will fail the condition, because their paths do not match your condition of *\AppData\Local\Microsoft\Terminal Server Client\Cache*.

Your subfolder condition and/or where the search starts need to not exclude the things between the start location and the location of the files you want to match.

3

Hi,

I'm searching from \\server\c$\users

4

Then the first folder it considers would be something like \\server\c$\users\Username (for some Username), which doesn't match the subfolder clause, causing it (and everything in it) to be skipped.

In other words, \\server\c$\users\Username does not match *\AppData\Local\Microsoft\Terminal Server Client\Cache*, so a subfolder clause with that condition would exclude it and everything below it.

I think using the slower path filter, instead of subfolder, will probably be faster in the long run in this case, unless you need to do this often.

The faster subfolder method is possible but you'd need to carefully define the filter so it matches every folder in between your starting point and the nested folders you're interested in.

You could also go for an intermediate option which considers everything up to a certain depth, and then only allows deeper folders if they match the desired pattern:

SubFolder
    Full-Path Match *\AppData\Local\Microsoft\Terminal Server Client\Cache*
    OR
    Full-Path No-Match \\server\c$\users\*\*\*\*\*\*

(Or something like that. You might need to experiment with the number of \* to get the depth you need.)

That would still consider a lot of folders that aren't needed, but would still exclude any deeper than you want which are outside of the area you want to search.

5

Thank you Leo,

I did a quick benchmark:
with only Full Path, in 20 seconds, it is still searching through users starting with "A".
with Full Path + your sub-folder filters, in 20 seconds it has reached users starting with "L".

I have the feeling this could be a lot quicker, but your proposition is way quicker than just the Full path.

As I need to do it regularly, and on multiple servers, if I want to get an almost instant result, would I need to call a script to build a collection ?
Is it possible to do something following this logic :
for each member of c:\users
then list content of c:\users\$member\AppData\Local\Microsoft\Terminal Server Client\Cache\
and add it to collection
?

While waiting for the search to finish, I was wondering if there is a way to display the amount of time it takes for a search to complete ? Has it been requested before ? This could be useful to display it, for instance in the status bar.

6

You won't need a script, but you might need a clause for each sub-folder between the start point and the end point.

There's no way to know that, without doing the search twice, since there's no way to know how many directories there are to read without reading the directories.

7

I'm not sure if I know how to do it.
I need to process and try to think how to do what you said...

I meant, once the search is completed, something to indicate how long it took.

8

Let's check if I understood the problem:

  • You start with a folder \\server\c$\users
  • This folder contains a rather large number of folders like user001, user002, ... user999
  • You are only interested in the contents of the folders
    user001\AppData\Local\Microsoft\Terminal Server Client\Cache\*.*
    user002\AppData\Local\Microsoft\Terminal Server Client\Cache\*.*
    ...
    user999\AppData\Local\Microsoft\Terminal Server Client\Cache\*.*
  • The result should be one collection

In that case I'd just use the Copy command:

Copy PATH="\\server\c$\users\{file}\AppData\Local\Microsoft\Terminal Server Client\Cache\*.*" CREATEFOLDER=coll:\\MyMegaSearch

You would need to select the folders user001, user002, ... user999 and run the button. A script could enumerate all folders automatically.

If I needed to use the Find command I'd use the same logic to create the paths for the IN part of the command.

In a script: yes, in a standard button: no

9

You understood perfectly the goal.

I tried the command you gave me, "coll:" takes slashed, not backslashes, that I could fix myself, however it only add something called "PATH" to the collection.

I had to remove the PATH= from the Copy command you gave me...
then it works if I select only 1 user. As soon as I select more, it treats only the first user and ignore others.

Any idea how to repeat the command on other users ?

10

Well spotted! Should have been FILE instead of PATH. Sorry for the confusion.

Looks like I was a bit too optimistic about the folders getting enumerated in a normal button.

Here is a little script that will do it:

function OnClick(clickData) {
    var cmd = clickData.func.command;
    var tab = clickData.func.sourcetab;
    cmd.deselect = false;

    cmd.RunCommand('Delete FILE="coll://MyMegaSearch" QUIET');

    for (var e = new Enumerator(tab.selected); !e.atEnd(); e.moveNext()) {
        var item = e.item();
        var cmdLine = 'Copy FILE="' + item + '\\AppData\\Local\\Microsoft\\Terminal Server Client\\Cache\\*.*" CREATEFOLDER="coll://MyMegaSearch"';
        DOpus.Output(cmdLine);
        cmd.RunCommand(cmdLine);
    }
}

35118.dcf (1.2 KB)

11

Just remembered the missing pieces: if you prefix the Copy command with @sync:dopusrt /cmd it should work as well without a script.

@nodeselect 
Delete FILE="coll://MyMegaSearch" QUIET
@sync:dopusrt /cmd Copy FILE="\\server\c$\users\{file}\AppData\Local\Microsoft\Terminal Server Client\Cache\*.*" CREATEFOLDER="coll://MyMegaSearch"
12

Hi,

It looks nice, I'm looking forward to trying it tomorrow morning. Thank you!

Is there a way to replace the need to select the folders? I'm only interested in selecting all folders under \users anyway.

Cheers

13

For a one-stop solution the script needs to be modified a bit:

function OnClick(clickData) {
    var cmd = clickData.func.command;
    var fsu = DOpus.FSUtil();
    cmd.deselect = false;

    var baseFolder = '\\\\server\\c$\\users';

    cmd.RunCommand('Delete FILE="coll://MyMegaSearch" QUIET');

    var folderEnum = fsu.ReadDir(baseFolder);
    while (!folderEnum.complete) {
        var item = folderEnum.Next();
        if (!item.is_dir) continue;
        var cmdLine = 'Copy FILE="' + item + '\\AppData\\Local\\Microsoft\\Terminal Server Client\\Cache\\*.*" CREATEFOLDER="coll://MyMegaSearch"';
        cmd.RunCommand(cmdLine);
    }

    cmd.RunCommand('Go PATH="coll://MyMegaSearch" NEWTAB=findexisting');
}

35118b.dcf (1.5 KB)

14

this is just perfect !
I'm going to create 1 button per server.

This was a good opportunity to learn more about DO, thank you for your script :slight_smile:

This would be nice if DO could "magically" do the same by evaluating wildcards in the middle of the path:
Copy FILE="\\server\c$\users\*\AppData\Local\Microsoft\Terminal Server Client\Cache\*.*" CREATEFOLDER="coll://MyMegaSearch"

@Leo ? Is it possible to consider this or would it be way too tricky for the dev ?