There is a bug in building the linked list of line records in the
diff iterator and also an off by one element error in the hunk
counts. This fixes both of these, adds some test data with more
complex sets of hunk and line diffs to exercise this code better.
This reduces the rate of syscalls for the common case of sequences of
object reads from the same pack.
Best of 5 timings for libgit2_clar before this patch:
real 0m5.375s
user 0m0.392s
sys 0m3.564s
After applying this patch:
real 0m5.285s
user 0m0.356s
sys 0m3.544s
0.6% improvement in system time.
9.2% improvement in user time.
1.7% improvement in elapsed time.
Confirmed a 0.6% reduction in number of system calls with strace.
Expect greater improvement for graph-traversal with large packs.
Fixed some minor `git_repository_hashfile` issues:
- Fixed incorrect doc (saying that repo could be NULL)
- Added checking of object type value to acceptable ones
- Added more tests for various parameter permutations
The existing `git_odb_hashfile` does not apply text filtering
rules because it doesn't have a repository context to evaluate
the correct rules to apply. This adds a new hashfile function
that will apply repository-specific filters (based on config,
attributes, and filename) before calculating the hash.
In the process of adding tests for the max file size threshold
(which treats files over a certain size as binary) there seem to
be a number of problems in the new code with detecting binaries.
This should fix those up, as well as add a test for the file
size threshold stuff.
Also, this un-deprecates `GIT_DIFF_LINE_ADD_EOFNL`, since I
finally found a legitimate situation where it would be returned.
Example: a cached node is owned only by the cache (refcount == 1).
Thread A holds the lock and determines that the entry which should get
cached equals the node (git_oid_cmp(&node->oid, &entry->oid) == 0).
It frees the given entry to instead return the cached node to the user
(entry = node). Now, before Thread A happens to increment the refcount
of the node *outside* the cache lock, Thread B tries to store another
entry and hits the slot of the node before, decrements its refcount and
frees it *before* Thread A gets a chance to increment for the user.
git_cached_obj_incref(entry);
git_mutex_lock(&cache->lock);
{
git_cached_obj *node = cache->nodes[hash & cache->size_mask];
if (node == NULL) {
cache->nodes[hash & cache->size_mask] = entry;
} else if (git_oid_cmp(&node->oid, &entry->oid) == 0) {
git_cached_obj_decref(entry, cache->free_obj);
entry = node;
} else {
git_cached_obj_decref(node, cache->free_obj);
// Thread B is here
cache->nodes[hash & cache->size_mask] = entry;
}
}
git_mutex_unlock(&cache->lock);
// Thread A is here
/* increase the refcount again, because we are
* returning it to the user */
git_cached_obj_incref(entry);
Often `git_odb_read_header` will "fail" and have to read the
entire object into memory instead of just the header. When this
happens, the object is loaded and then disposed of immediately,
which makes it difficult to efficiently use the header information
to decide if the object should be loaded (since attempting to do
so will often result in loading the object twice).
This commit takes the existing code and reorganizes it to have
two new functions:
- `git_odb__read_header_or_object` which acts just like the old
read header function except that it returns the object, too, if
it was forced to load the whole thing. It then becomes the
callers responsibility to free the `git_odb_object`.
- `git_object__from_odb_object` which was extracted from the old
`git_object_lookup` and creates a subclass of `git_object` from
an existing `git_odb_object` (separating the ODB lookup from the
`git_object` creation). This allows you to use the first header
reading function efficiently without instantiating the
`git_odb_object` twice.
There is no net change to the behavior of any of the existing
functions, but this allows internal code to tap into the ODB
lookup and object creation to be more efficient.
This commit adds a max_size value in the public `git_diff_options`
structure so that the user can automatically flag blobs over a
certain size as binary regardless of other properties.
Also, and perhaps more importantly, this moves binary detection
to be as early as possible in the diff traversal inner loop and
makes sure that we stop loading objects as soon as we decide that
they are binary.
The `git_diff_iterator_num_files` API was problematic, since we
don't actually know the exact number of files to be iterated over
until we load those files into memory. This replaces it with a
new `git_diff_iterator_progress` API that goes from 0 to 1, and
moves and renamed the old API for the internal places that can
tolerate a max value instead of an exact value.
Previously when diffing blobs, the diff code just ran with a NULL
repository object. Of course, that's not necessary and the test
for a NULL repo was confusing. This makes the blob diff run with
the repo that contains the blobs and clarifies the test that it
is possible to be diffing data where the path is unknown.
This adds support to diff and status for running filters (a la crlf)
on blobs in the workdir before computing SHAs and before generating
text diffs. This ended up being a bit more code change than I had
thought since I had to reorganize some of the diff logic to minimize
peak memory use when filtering blobs in a diff.
This also adds a cap on the maximum size of data that will be loaded
to diff. I set it at 512Mb which should match core git. Right now
it is a #define in src/diff.h but it could be moved into the public
API if desired.
This refactors the diff output code so that an iterator object
can be used to traverse and generate the diffs, instead of just
the `foreach()` style with callbacks. The code has been rearranged
so that the two styles can still share most functions.
This also replaces `GIT_REVWALKOVER` with `GIT_ITEROVER` and uses
that as a common error code for marking the end of iteration when
using a iterator style of object.
SSL_get_error() allows to receive a result code for various SSL
operations. Depending on the return value (see man (3) SSL_get_error)
there might be additional information in the OpenSSL error queue. Return
the queued message if available, otherwise set an error message
corresponding to the return code.
There is a better and less fragile way to calculate time offsets. Let
the OS take care of dealing with DST and simply take the the offset
between the local time and UTC that it gives us.
Passing SSL_VERIFY_PEER makes OpenSSL shut down the connection if the
certificate is invalid, without giving us a chance to ignore that
error. Pass SSL_VERIFY_NONE and call SSL_get_verify_result if the user
wanted us to check.
When no CNs match, we used to jump to on_error which gave a bogus
error as that's for OpenSSL errors. Jump to cert_fail so we tell the
user that the error came from checking the certificate.
This expands the types of peeling that `git_object_peel` knows
how to do to include TAG -> BLOB peeling, and makes the errors
slightly more consistent depending on the situation. It also
adds a new special behavior where peeling to ANY will peel until
the object type changes (e.g. chases TAGs to a non-TAG).
Using this expanded peeling, this replaces peeling code that was
embedded in `git_tag_peel` and `git_reset`.
It's not really needed with the current code as we have EOS and the
sideband's flush to tell us we're done.
Keep the distinction between processed and received objects.
Just clean up valgrind warnings about uninitialized memory
and also clear out errno in some cases where it results in
a false error message being generated at a later point.
This is a big redesign of the git_submodule_status API and the
implementation of the redesigned API. It also fixes a number of
bugs that I found in other parts of the submodule API while
writing the tests for the status part.
This also fixes a couple of bugs in the iterators that had not
been noticed before - one with iterating when there is a gitlink
(i.e. separate-work-dir) and one where I was treating anything
even vaguely submodule-like as a submodule, more aggressively
than core git does.
This fixes up a number of problems flagged by valgrind and also
cleans up the internal `git_submodule` allocation handling
overall with a simpler model.
This cleans up a number of items suggested during code review
with @vmg, including:
* renaming "outside repo" config API to `git_config_open_default`
* killing the `git_config_open_global` API
* removing the `git_` prefix from the static functions in fileops
* removing some unnecessary functionality from the "cp" command
This extends git_repository_init_ext further with support for
initializing the repository from an external template directory
and with support for the "create shared" type flags that make a
set GID repository directory.
This also adds tests for much of the new functionality to the
existing `repo/init.c` test suite.
Also, this adds a bunch of new utility functions including a
very general purpose `git_futils_mkdir` (with the ability to
make paths and to chmod the paths post-creation) and a file
tree copying function `git_futils_cp_r`. Also, this includes
some new path functions that were useful to keep the code
simple.
The extended version of repository init adds support for many
of the things that you can do with `git init` and sets up
structures that will make it easier to extend further in the
future.
In looking at PR #878, I found a few small bugs in the diff code,
mostly related to work that can be avoided when processing tree-
to-tree diffs that was always being carried out. This commit has
some small fixes in it.
This creates a public API for adding to the internal ignores
list, which already existing but was not accessible.
This adds the new default value for core.excludesfile also.
Up to now, the idea was that the user would do all the operations for
one repository in the same thread. Thus we could have the
memory-mapped window information thread-local and avoid any locking.
This is not practical in a few environments, such as Apple's GCD which
allocates threads arbitrarily or the .NET CLR, where the OS-level
thread can change at any moment.
Make the control structure global and protect it with a mutex so we
don't depend on the thread currently executing the code.
If you want to be absolutely safe with git_message_prettify, you
can now pass a NULL pointer for the buffer and get back the number
of bytes that would be copied into the buffer.
This means that an error is a non-negative return code and a
success will be greater than zero from this function.
Returning a negative cancels the walk, and returning a positive one
causes us to skip an entry, which was previously done by a negative
value.
This allows us to stay consistent with the rest of the functions that
take a callback and keeps the skipping functionality.
There is a little cleanup necessary from PR #843. Since the
new callbacks return `GIT_EUSER` we have to be a little careful
about return values when they are used internally to the library.
Also, callbacks should be checked for non-zero return values,
not just less than zero.
This updates all the `foreach()` type functions across the library
that take callbacks from the user to have a consistent behavior.
The rules are:
* A callback terminates the loop by returning any non-zero value
* Once the callback returns non-zero, it will not be called again
(i.e. the loop stops all iteration regardless of state)
* If the callback returns non-zero, the parent fn returns GIT_EUSER
* Although the parent returns GIT_EUSER, no error will be set in
the library and `giterr_last()` will return NULL if called.
This commit makes those changes across the library and adds tests
for most of the iteration APIs to make sure that they follow the
above rules.
Fixes#824
Exporting variables in a dynamic library is a PITA. Let's keep
these values internally and wrap them through a helper method.
This doesn't break the external API. @arrbee, aren't you glad I turned
the `GIT_ATTR_` macros into function macros? ✨
The 'git revert/cherry-pick/merge -n' commands leave .git/MERGE_MSG
behind so that git-commit can find it. As we don't yet support these
operations, users who are shelling out to let git perform these
operations haven't had a convenient way to get this message.
These functions allow the user to retrieve the message and remove it
when she's created the commit.
Instad of each transport having its own function and logic to get to
its refs, store them directly in transport.
Leverage the new gitno_buffer to make the parsing and storing of the
refs use common code and get rid of the git_protocol struct.
This allows us to add capabilitites to both at the same time, keeps
them in sync and removes a lot of code.
gitno_buffer now uses a callback to fill its buffer, allowing us to
use the same interface for git and http (which uses callbacks).
For the transition, http is going to keep its own logic until the
git/common code catches up with the implied multi_ack that http
has. This also has the side-effect of making the code cleaner and more
correct regardingt he protocol.
git.git uses an inlined hashcmp function instead of memcmp, since it
performes much better when comparing hashes (most hashes compared
diverge within the first byte).
Measurements and rationale for the curious reader:
http://thread.gmane.org/gmane.comp.version-control.git/172286
Renamed git_checkout_index to what it really was,
and removed duplicate code from clone.c. Added
git_checkout_ref, which updates HEAD and hands off
to git_checkout_head.
Added tests for the options the caller can pass to
git_checkout_*.
This makes sure that an error code returned by the callback function
of `git_tree_walk` will stop the iteration and get propagated back
to the caller verbatim.
Also, this adds a minor helper function `git_tree_entry_byoid` that
searches a `git_tree` for an entry with the given OID. This isn't
a fast function, but it's easier than writing the loop yourself as
an external user of the library.
A diff that is created with a NULL options parameter could result
in a NULL prefix string, but diff merge was unconditionally
strdup'ing it. I added a test to replicate the issue and then a
new method that does the right thing with NULL values.
Josh Triplett noticed libgit2 actually does preorder entries in
tree_walk_post instead of postorder. Also, we continued walking even
when an error occured in the callback.
Fix#773; also, allow both pre- and postorder walking.
Also removes the unnecessary check for filter
length, since git_filters_apply does the right
thing when there are none, and it's more efficient
than this.
Not all delta bases are available on the first try. By delaying
resolving all deltas until the end, we avoid decompressing some of the
data twice or even more times, saving effort and time.
The correct way to advertise out capabilities is by appending them to
the first 'want' line, using SP as separator, instead of NUL as the
server does. Inconsistent documentation lead to the use of NUL in
libgit2.
Fix this so we can request much more efficient packs from the
remote which reduces the indexing time considerably.
passing 0 to git_strol(32|64) let the implementation guess if it's
dealing with an octal number or a decimal one.
Let's make it safe and ensure that both 'HEAD@{010}' and 'HEAD@{10}'
point at the same commit.
This added a flag to the `git_repository_set_workdir()` function
that enables generation of a `.git` gitlink file that links the
new workdir to the parent repository. Essentially, the flag tells
the function to write out the changes to disk to permanently set
the workdir of the repository to the new path.
If you pass this flag as true, then setting the workdir to something
other than the default workdir (i.e. the parent of the .git repo
directory), will create a plain file named ".git" with the standard
gitlink contents "gitdir: <repo-path>", and also update the
"core.worktree" and "core.bare" config values.
Setting the workdir to the default repo workdir will clear the
core.worktree flag (but still permanently set core.bare to false).
BTW, the libgit2 API does not currently provide a function for
clearing the workdir and converting a non-bare repo into a bare one.
Adding a new config iteration function that let's you iterate
over just the config entries that match a particular regular
expression. The old foreach becomes a simple use of this with
an empty pattern.
This also fixes an apparent bug in the existing `git_config_foreach`
where returning a non-zero value from the iteration callback was
not correctly aborting the iteration and the returned value was
not being propogated back to the caller of foreach.
Added to tests to cover all these changes.
This makes it easy to take a buffer containing a path with relative
references (i.e. .. or . path segments) and resolve all of those
into a clean path. This can be applied to URLs as well as file
paths which can be useful.
As part of this, I made the drive-letter detection apply on all
platforms, not just windows. If you give a path that looks like
"c:/..." on any platform, it seems like we might as well detect
that as a rooted path. I suppose if you create a directory named
"x:" on another platform and want to use that as the beginning
of a relative path under the root directory of your repo, this
could cause a problem, but then it seems like you're asking for
trouble.
* `git_buf_rfind` (with tests and tests for `git_buf_rfind_next`)
* `git_buf_puts_escaped` and `git_buf_puts_escaped_regex` (with tests)
to copy strings into a buffer while injecting an escape sequence
(e.g. '\') in front of particular characters.
