fscache: add not-found directory cache to fscache #994
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jeffhostetler
force-pushed
the
jeffhostetler/fscache_nfd
branch
from
e6e29d2
to
d32841b
Compare
jeffhostetler
changed the title
jeffhostetler
changed the title
|
Ooh, this should be a pretty decent performance improvement when using sparse. 😁 |
dscho
reviewed
Dec 22, 2016
| @@ -6,8 +6,83 @@ | |||
| static int initialized; | |||
| static volatile long enabled; | |||
| static struct hashmap map; | |||
| static struct hashmap map_nfd; /* not found directories */ | |||
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
|
|
||
| static struct nfd_entry *nfd_alloc(const char *name, size_t namelen, unsigned int hash) | ||
| { | ||
| struct nfd_entry *nfd = xcalloc(1, sizeof(struct nfd_entry) + namelen + 1); |
This comment was marked as off-topic.
This comment was marked as off-topic.
Sorry, something went wrong.
|
@dscho I pulled your fixup and added one more. |
|
@jeffhostetler okay, good. Please note that the Other than that, I think the only thing we may want to consider is to try our hand at a test that verifies somehow that non-existing directories are not accessed more than once. We could introduce a new What do you think? |
Brilliant. The more optional tracing we enable, the easier debugging will be in the future. |
|
@dscho Yeah. I'll re-title the commit and look at adding the tracing. I'll try to fix this up next week. Thanks! |
Well, we should not overdo it, in particular in performance-critical code such as FSCache... 😄 |
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach FSCACHE to remember "not found" directories. This is a performance optimization. FSCACHE is a performance optimization available for Windows. It intercepts Posix-style lstat() calls into an in-memory directory using FindFirst/FindNext. It improves performance on Windows by catching the first lstat() call in a directory, using FindFirst/ FindNext to read the list of files (and attribute data) for the entire directory into the cache, and short-cut subsequent lstat() calls in the same directory. This gives a major performance boost on Windows. However, it does not remember "not found" directories. When STATUS runs and there are missing directories, the lstat() interception fails to find the parent directory and simply return ENOENT for the file -- it does not remember that the FindFirst on the directory failed. Thus subsequent lstat() calls in the same directory, each re-attempt the FindFirst. This completely defeats any performance gains. This can be seen by doing a sparse-checkout on a large repo and then doing a read-tree to reset the skip-worktree bits and then running status. This change reduced status times for my very large repo by 60%. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
jeffhostetler
force-pushed
the
jeffhostetler/fscache_nfd
branch
from
1f78c19
to
8544ec3
Compare
|
In this version, I added a simple GIT_TRACE_FSCACHE key to log each time FindFirst fails. |
|
Excellent! I added a test that I merged, too (10b99b6). |
dscho
added a commit
to git-for-windows/build-extra
that referenced
this pull request
Jan 25, 2017
Performance [was enhanced when using fscache in a massively sparse checkout](git-for-windows/git#994). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
dscho
added a commit
to dscho/git
that referenced
this pull request
Feb 1, 2017
…er/fscache_nfd fscache: add not-found directory cache to fscache
dscho
added a commit
that referenced
this pull request
Feb 2, 2017
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Feb 2, 2017
fscache: add not-found directory cache to fscache
dscho
added a commit
that referenced
this pull request
Feb 2, 2017
fscache: add not-found directory cache to fscache
dscho
added a commit
that referenced
this pull request
Feb 2, 2017
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Feb 2, 2017
fscache: add not-found directory cache to fscache
dscho
added a commit
that referenced
this pull request
Feb 3, 2017
fscache: add not-found directory cache to fscache
dscho
added a commit
that referenced
this pull request
Feb 4, 2017
fscache: add not-found directory cache to fscache
dscho
added a commit
that referenced
this pull request
Feb 4, 2017
fscache: add not-found directory cache to fscache
dscho
added a commit
that referenced
this pull request
Feb 4, 2017
fscache: add not-found directory cache to fscache
dscho
added a commit
to dscho/git
that referenced
this pull request
Feb 4, 2017
…er/fscache_nfd fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 22, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 22, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 23, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 23, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 24, 2024
fscache: add not-found directory cache to fscache
dscho
added a commit
to microsoft/git
that referenced
this pull request
Apr 24, 2024
…er/fscache_nfd fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 24, 2024
fscache: add not-found directory cache to fscache
dscho
added a commit
that referenced
this pull request
Apr 25, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 25, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 25, 2024
fscache: add not-found directory cache to fscache
dscho
added a commit
that referenced
this pull request
Apr 25, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 25, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 26, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 26, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 26, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 26, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 26, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 27, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 27, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 29, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 29, 2024
fscache: add not-found directory cache to fscache
dscho
added a commit
to microsoft/git
that referenced
this pull request
Apr 29, 2024
…er/fscache_nfd fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 29, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
Apr 29, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
May 1, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
May 1, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
May 1, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
May 4, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
May 4, 2024
fscache: add not-found directory cache to fscache
git-for-windows-ci
pushed a commit
that referenced
this pull request
May 9, 2024
fscache: add not-found directory cache to fscache
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Teach FSCACHE to remember "not found" directories.
This is a performance optimization.
FSCACHE is a performance optimization available for Windows. It
intercepts Posix-style lstat() calls into an in-memory directory
using FindFirst/FindNext. It improves performance on Windows by
catching the first lstat() call in a directory, using FindFirst/
FindNext to read the list of files (and attribute data) for the
entire directory into the cache, and short-cut subsequent lstat()
calls in the same directory. This gives a major performance
boost on Windows.
However, it does not remember "not found" directories. When STATUS
runs and there are missing directories, the lstat() interception
fails to find the parent directory and simply return ENOENT for the
file -- it does not remember that the FindFirst on the directory
failed. Thus subsequent lstat() calls in the same directory, each
re-attempt the FindFirst. This completely defeats any performance
gains.
This can be seen by doing a sparse-checkout on a large repo and
then doing a read-tree to reset the skip-worktree bits and then
running status.
This change reduced status times for my very large repo by 60%.