Deduplicate and optimize file index

[termermc]

Right now, if you create a share of /mnt/cat_videos on two different servers, you will end up having two different indexes for the same folder, one per-share.

I propose that indexes be keyed by a short hash of their path on disk + the index settings. That way, if two shares with the same indeex settings point to /mnt/cat_videos, the directory will only be indexed once. But if there are different settings (like symlinks being followed on one but not the other), they will get their own indexes to ensure that file privacy is preserved.

I was also thinking of just keying indexes by their files and then using a prefix search, but I don't think that could be used if we have different shares with different index settings.

[termermc]

Since indexes and shares need to follow the same logic (like respecting no symlinks), I think we should have a central FS management abstraction, where you request a filesystem reference for the specified type, path, and options (like local, /home/me/Music, follow symlinks), and then you get a filesystem abstraction that lets you get files and read them, similar to what the Share does now. And the FS manager can manage indexes. You can create a hash from a reference based on the type, path, options.

Also, I'd like to move indexes to their own SQLite databases, one per FS hash. That way, we don't have contention and bloat on the main client.db. Instead, it could be index/<hash>.db. When doing a full re-index, we could also atomically swap the files and delete the old one, and replace the SQLite connection (with locking of course).

[termermc]

This could be the basis for supporting other filesystem types, like S3-compatible, SFTP, etc. in the future.