When doing a diff for use in status, we should never show the
content of a git repository contained inside another one. The
logic to do this was looking for a .git directory and so when a
gitlink plain .git file was used, it was failing to exclude the
directory content.
There was a little bug where the submodule cache thought that the
index date was out of date even when it wasn't that was resulting
in some extra scans of index data even when not needed.
Mostly this commit adds a bunch of new tests including adding and
removing submodules in the index and in the HEAD and seeing if we
can automatically pick them up when refreshing.
Wrote tests that try adding, removing, and updating the name of
submodules which showed a number of problems with how we account
for changes when incrementally updating the submodule info. Most
of these issues didn't exist before because reloading would always
blow away the old submodule data.
This improvement the management of the lock around submodule cache
updates slightly, using the lock to make sure that foreach can
safely make a snapshot of all existing submodules and making sure
that git_submodule_add_setup also grabs a lock before inserting
the new submodule. Cache initialization / refresh should already
have been holding the lock correctly as it adds submodules.
When forcing cache flushes or reload, etc., it is easier to keep
track of intent using enums instead of plain bools. Also, this
fixes a bug where the cache was not being properly refreshes by
a git_submodule_reload_all.
This makes submodule cache refresh actually look at the timestamps
from the data sources for submodules and reload as needed if they
have changed since the last refresh.
This takes the old submodule cache which was just a git_strmap
and makes a real git_submodule_cache object that can contain other
things like a lock and timestamp-ish data to control refreshing of
submodule info.
This fixes `git_submodule_sync` to correctly update the remote URL
of the default branch of the submodule along with the URL in the
parent repository config (i.e. match core Git's behavior).
Also move some useful helper logic from the submodule code into
a shared config API `git_config__update_entry` that can either set
or delete an entry with constraints like not overwriting or not
creating a new entry. I used that helper to update a couple other
places in the code.
There are a few places where we need to join three strings to
assemble a path. This adds a simple join3 function to avoid the
comparatively expensive join_n (which calls strlen on each string
twice).
The order in this function is the opposite to what
create_with_fetchspec() has, so change this one, as url-then-refspec is
what git does.
As we need to break compilation and the swap doesn't do that, let's take
this opportunity to rename in-memory remotes to anonymous as that's
really what sets them apart.
With the changes to how git_path_dirload_with_stat handles things
that look like submodules, submodules could end up sorted in the
wrong order with the workdir iterator. This moves the submodule
check earlier in the iterator processing of a new directory so
that the submodule name updates will happen immediately and the
sort order will be correct.
While looping over multiple heads, an up-to-date head will clobber the `remote->need_pack` setting, preventing the rest of the machinery from building and downloading a pack-file, breaking fetches.
If the first call to release a no-longer-existent submodule freed
the object, the check if a second is needed would dereference the
data that was just freed.
When a file is open for reading (without shared-delete permission), and
then a different thread/process called p_rename, that would fail, even
if the file was only open for reading for a few milliseconds. This
change lets p_rename wait up to 50ms for the file to be closed by the
reader. Applies only to win32.
This is especially important for git_filebuf_commit, because writes
should not fail if the file is read simultaneously.
Fixes#2207
When a submodule was inserted with a different path and name, the
return value from khash greater than zero was allowed to propagate
back out to the caller when it should really be zeroed. This led
to a possible crash when reloading submodules if that was the
first time that submodule data was loaded.
The reload_all call could end up dereferencing a NULL pointer if
there was an error while attempting to load the submodules config
data (i.e. invalid content in the gitmodules file). This fixes it.
This cleans up some places I missed that could hold onto submodule
references and cleans up the way in which the repository cache is
both reloaded and released so that existing submodule references
aren't destroyed inappropriately.
When a directory containing a .git directory (or even just a plain
gitlink) was found, libgit2 was going out of its way to treat it
specially. This seemed like it was necessary because the diff
code was not originally emulating Git's behavior for untracked
directories correctly (i.e. scanning for ignored vs untracked items
inside). Now that libgit2 diff mimics Git's untracked directory
behavior, the special handling for contained Git repos is actually
incorrect and this commit rips it out.
`git_submodule` objects were already refcounted internally in case
the submodule name was different from the path at which it was
stored. This makes that refcounting externally used as well, so
`git_submodule_lookup` and `git_submodule_add_setup` return an
object that requires a `git_submodule_free` when done.
As a way to speed up the cases where we need to hide some commits, we
find out what the merge bases are so we know to stop marking commits as
uninteresting and avoid walking down a potentially very large amount of
commits which we will never see. There are however two oversights in
current code.
The merge-base finding algorithm fails to recognize that if it is only
given one commit, there can be no merge base. It instead walks down the
whole ancestor chain needlessly. Make it return an empty list
immediately in this situation.
The revwalk does not know whether the user has asked to hide any commits
at all. In situation where the user pushes multiple commits but doesn't
hide any, the above fix wouldn't do the trick. Keep track of whether the
user wants to hide any commits and only run the merge-base finding
algorithm when it's needed.
The reflog append function was overzealous in its checking. When passed
an old and new ids, it should not do any checking, but just serialize
the data to a reflog entry.
The existing ones lack checking zeroed ids when switching back from an
unborn branch as well as what happens when detaching.
The reflog appending function mistakenly wrote zeros when dealing with a
detached HEAD. This explicitly checks for those situations and fixes
them.
When we update the current branch, we must also append to HEAD's reflog
to keep them in sync.
This is a bit of a hack, but as git.git says, it covers 100% of
default cases.
If the pqueue comparison fn returned just 0 or 1 (think "a<b")
then the sort order of returned items could be wrong because there
was a "< 0" that really needed to be "<= 0". Yikes!!!
The git_odb_exists_prefix API was not dealing correctly when a
later backend returned GIT_ENOTFOUND even if an earlier backend
had found the object.
Additionally, the unit tests were not properly exercising the API
and had a couple mistakes in checking the results.
Lastly, since the backends are not expected to behavior correctly
unless all bytes of the short id are zero except for the prefix,
this makes the ODB prefix APIs explicitly clear out the extra
bytes so the user doesn't have to be as careful.
We look up a reference in order to figure out if it's the current
branch, which we need to free once we're done with the check.
As a bonus, only perform the check when we're passed the force flag, as
it's a useless check otherwise.
We can make use of git_object_dup to use refcounting instead of pointer
comparison to make sure we don't free the caller's object.
This also lets us simplify the case for '~0' which is now just an
assignment instead of looking up the object we have at hand.
Previously the hunk_byfinalline_search_cmp function was called with different
data types (size_t and uint32_t) for the key argument but expected only the
former resulting in an invalid memory access when passed the latter on a 64 bit
machine.
The following patch makes sure that the function is called and works with the
same type (size_t).
If a directory disappears between the time we look up the entries of its
parent and the time when we go to look at it, we should ignore the error
and move forward.
This fixes#2046.
This finds a short id string that will unambiguously select the
given object, starting with the core.abbrev length (usually 7)
and growing until it is no longer ambiguous.
A few fixes have accumulated in this area which have made the freeing of
data a bit muddy. Make sure to free the data only when needed and once.
When we are going to write a delta to the packfile, we need to free the
data, otherwise leave it. The current version of the code mixes up the
checks for po->data and po->delta_data.
This adds `git_diff_buffers` and `git_patch_from_buffers`. This
also includes a bunch of internal refactoring to increase the
shared code between these functions and the blob-to-blob and
blob-to-buffer APIs, as well as some higher level assert helpers
in the tests to also remove redundancy.
- added MSVC cmake definitions to disable warnings
- general.c is rewritten so it is ansi-c compatible and compiles ok on microsoft windows
- some MSVC reported warning fixes
* Make GIT_INLINE an internal definition so it cannot be used in
public headers
* Fix language in CONTRIBUTING
* Make index caps API use signed instead of unsigned values
On some systems, notably HP PA-RISC systems running Linux or HP-UX,
EWOULDBLOCK and EAGAIN are not the same value. POSIX (and these OSes) allow
EWOULDBLOCK to occur on write(2) (and send(2), etc.), so check explicitly
for this case as well as EAGAIN by defining and using a macro GIT_ISBLOCKED
that considers both.
The macro is necessary because MSYS does not provide EWOULDBLOCK and
compilation fails if an attempt is made to use it unconditionally. On most
systems, where the two values are the same, the compiler will simply
optimize this check out and it will have no effect.
This adds an API to amend an existing commit, basically a shorthand
for creating a new commit filling in missing parameters from the
values of an existing commit. As part of this, I also added a new
"sys" API to create a commit using a callback to get the parents.
This allowed me to rewrite all the other commit creation APIs so
that temporary allocations are no longer needed.
This fixes a number of warnings with the Windows 64-bit build
including a test failure in test_repo_message__message where an
invalid pointer to a git_buf was being used.
We need this from util.h and posix.h, but the latter includes common.h
which includes util.h, which means p_strlen is not defined by the time
we get to git__strndup().
Split the definition on p_strlen() off into its own header so we can use
it in util.h.
The current code issues a lot of strncmp() calls in order to check for
the end of the header, simply in order to copy it and start going
through it again. These are a lot of calls for something we can check as
we go along. Knowing the amount of parents beforehand to reduce
allocations in extreme cases does not make up for them.
Instead start parsing immediately and check for the double-newline after
each header field, leaving the raw_header allocation for the end, which
lets us go through the header once and reduces the amount of strncmp()
calls significantly.
In unscientific testing, this has reduced a shortlog-like usage (walking
though the whole history of a branch and extracting data from the
commits) of git.git from ~830ms to ~700ms and makes the time we spend in
strncmp() negligible.
Let the user push committish objects and peel them to figure out which
commit to push to our queue.
This is for convenience and for allowing uses of
git_revwalk_push_glob(w, "tags")
with annotated tags.
Change the name to _matching() intead of _if(), and force _set_target()
to be a conditional update. If the user doesn't care about the old
value, they should use git_reference_create().
This tweaks the pqueue_up and pqueue_down routines so that they
will not do full element swaps but instead carry over the state
of the previous loop iteration and only assign elements for which
we know the final position. This will avoid a little bit of data
assignment which should improve performance in theory.
Also got rid of some vector helpers that I'm no longer using.
This fixes a typo I made for setting the sorted flag on the index
after a reload. That typo didn't actually cause any test failures
so I'm also adding a test that explicitly checks that the index is
correctly sorted after a reload when ignoring case and when not.
This updates the git_pqueue to simply be a set of specialized
init/insert/pop functions on a git_vector.
To preserve the pqueue feature of having a fixed size heap, I
converted the "sorted" field in git_vectors to a more general
"flags" field so that pqueue could mix in it's own flag. This
had a bunch of ramifications because a number of places were
directly looking at the vector "sorted" field - I added a couple
new git_vector helpers (is_sorted, set_sorted) so the specific
representation of this information could be abstracted.
I accidentally wrote a separate priority queue implementation when
I was working on file rename detection as part of the file hash
signature calculation code. To simplify licensing terms, I just
adapted that to a general purpose priority queue and replace the
old priority queue implementation that was borrowed from elsewhere.
This also removes parts of the COPYING document that no longer
apply to libgit2.
Validating the workdir should not compare HEAD to working
directory - this is both inefficient (as it ignores the cache)
and incorrect. If we had legitimately allowed changes in the
index (identical to the merge result) then comparing HEAD to
workdir would reject these changes as different. Further, this
will identify files that were filtered strangely as modified,
while testing with the cache would prevent this.
Also, it's stupid slow.
The checkout code used to defer removal of "blocking" files in
checkouts until the blocked item was actually being written (since
we have already checked that the removing the block is acceptable
according to the update rules). Unfortunately, this resulted in
an intermediate index state where both the blocking and new items
were in the index which is no longer allowed. Now we just remove
the blocking item in the first pass so it never needs to coexist.
In cases where there are typechanges, this could result in a bit
more churn of removing and recreating intermediate directories,
but I'm going to assume that is an unusual case and the churn will
not be too costly.
There were some confusing issues mixing up the number of bytes
written to the zstream output buffer with the number of bytes
consumed from the zstream input. This reorganizes the zstream
API and makes it easier to deflate an arbitrarily large input
while still using a fixed size output.
This removes the fetchRecurse compiler warnings and makes the
behavior match the other submodule options (i.e. the in-memory
setting can be reset to the on-disk value).
When three-way merging indexes, we previously changed each path
as we read them, which would lead to us adding an index entry for
'foo', then removing an index entry for 'foo/file'. With the new
index requirements, this is not allowed. Removing entries in the
merged index, then adding them, resolves this. In the previous
example, we now remove 'foo/file' before adding 'foo'.
In case insensitive index mode, we would stop at a prefixed entry,
treating the provided search key length as a substring, not the
length of the string to match.
Writing a sample Javascript driver pointed out some extra
whitespace handling that needed to be done in the diff driver.
This adds some tests with some sample javascript code that I
pulled off of GitHub just to see what would happen. Also, to
clean up the userdiff test data, I did a "git gc" and packed
up the test objects.
Since I don't have permission yet on the code from Git, I decided
I'd take a stab at writing patterns for PHP and Javascript myself.
I think these are pretty weak, but probably better than the
default behavior without them.
I contacted a number of Git authors and lined up their permission
to relicense their work for use in libgit2 and copied over their
code for diff driver xfuncname patterns. At this point, the code
I've copied is taken verbatim from core Git although Thomas Rast
warned me that the C++ patterns, at least, really need an update.
I've left off patterns where I don't feel like I have permission
at this point until I hear from more authors.
Reorganize the builtin driver table slightly so that core Git
builtin definitions can be imported verbatim. Then take a few of
the core Git drivers and pull them in.
This also creates a test of diffs with the builtin HTML driver
which led to some small error handling fixes in the driver
selection logic.
This extends the diff driver parser to support multiline driver
definitions along with ! prefixing for negated matches. This
brings the driver function pattern parsing in line with core Git.
This also adds an internal table of driver definitions and a
fallback code path that will look in that table for diff drivers
that are set with attributes without having a definition in the
config file. Right now, I just populated the table with a kind
of simple HTML definition that is similar to the core Git def.
Don't try to determine whether the system supports file modes
when putting the tree data in the index during checkout. The tree's
mode is canonical and did not come from stat(2) in the first place.
It's hard or even impossible to correctly free the string buffer
allocated by git_patch_to_str in some circumstances. Drop the function
so people have to use git_patch_to_buf instead - git_buf has a dedicated
destructor.
Returning library-allocated strings from libgit2 works fine on Linux,
but may cause problems on Windows because there is no one C Runtime that
everything links against. With libgit2 not exposing its own allocator,
freeing the string is a gamble.
git_patch_to_str already serializes to a buffer, then returns the
underlying memory. Expose the functionality directly, so callers can use
the git_buf_free function to free the memory later.
The "merge none" (don't automerge) flag was only to aide in
merge trivial tests. We can easily determine whether merge
trivial resulted in a trivial merge or an automerge by examining
the REUC after automerge has completed.
The default merge_file level was XDL_MERGE_MINIMAL, which will
produce conflicts where there should not be in the case where
both sides were changed identically. Change the defaults to be
more aggressive (XDL_MERGE_ZEALOUS) which will more aggressively
compress non-conflicts. This matches git.git's defaults.
Increase testing around reverting a previously reverted commit to
illustrate this problem.
Any well-behaved program should write a descriptive message to the
reflog whenever it updates a reference. Let's make this more prominent
by removing the version without the reflog parameters.
This changes git_signature_dup to actually honor oom conditions raised by
the call to git__strdup. It also aligns it with the error code return
pattern used everywhere else.
Ok, scrap the previous commit. This is the right overflow check that
takes care of 64 bit overflow **and** 32-bit overflow, which needs to be
considered because the pool malloc can only allocate 32-bit elements in
one go.
Note that `git_pool_strdup` cannot really return any error codes,
because the pool doesn't set errors on OOM.
The only place where `giterr_set_oom` is called is in
`git_pool_strndup`, in a conditional check that is always optimized
away. `n + 1` cannot be zero if `n` is unsigned because the compiler
doesn't take wraparound into account.
This check has been removed altogether because `size_t` is not
particularly going to overflow.
This renames git_vector_free_all to the better git_vector_free_deep
and also contains a couple of memory leak fixes based on valgrind
checks. The fixes are specifically: failure to free global dir
path variables when not compiled with threading on and failure to
free filters from the filter registry that had not be initialized
fully.
This adds tests that try canceling an indexer operation from
within the progress callback.
After writing the tests, I wanted to run this under valgrind and
had a number of errors in that situation because mmap wasn't
working. I added a CMake option to force emulation of mmap and
consolidated the Amiga-specific code into that new place (so we
don't actually need separate Amiga code now, just have to turn on
-DNO_MMAP).
Additionally, I made the indexer code propagate error codes more
reliably than it used to.
Clone callbacks can return non-zero values to cancel the clone.
This adds some tests to verify that this actually works and updates
the documentation to be clearer that this can happen and that the
return value will be propagated back by the clone function.
The checkout notify callback behavior on non-zero return values
was not being tested. This adds tests, fixes a bug with positive
values, and clarifies the documentation to make it clear that the
checkout can be canceled via this mechanism.
The callback to supply data chunks could return a negative value
to stop creation of the blob, but we were neither using GIT_EUSER
nor propagating the return value. This makes things use the new
behavior of returning the negative value back to the user.
This changes the behavior of callbacks so that the callback error
code is not converted into GIT_EUSER and instead we propagate the
return value through to the caller. Instead of using the
giterr_capture and giterr_restore functions, we now rely on all
functions to pass back the return value from a callback.
To avoid having a return value with no error message, the user
can call the public giterr_set_str or some such function to set
an error message. There is a new helper 'giterr_set_callback'
that functions can invoke after making a callback which ensures
that some error message was set in case the callback did not set
one.
In places where the sign of the callback return value is
meaningful (e.g. positive to skip, negative to abort), only the
negative values are returned back to the caller, obviously, since
the other values allow for continuing the loop.
The hardest parts of this were in the checkout code where positive
return values were overloaded as meaningful values for checkout.
I fixed this by adding an output parameter to many of the internal
checkout functions and removing the overload. This added some
code, but it is probably a better implementation.
There is some funkiness in the network code where user provided
callbacks could be returning a positive or a negative value and
we want to rely on that to cancel the loop. There are still a
couple places where an user error might get turned into GIT_EUSER
there, I think, though none exercised by the tests.
There are a lot of places that we call git__free on each item in
a vector and then call git_vector_free on the vector itself. This
just wraps that up into one convenient helper function.
This continues auditing all the places where GIT_EUSER is being
returned and making sure to clear any existing error using the
new giterr_user_cancel helper. As a result, places that relied
on intercepting GIT_EUSER but having the old error preserved also
needed to be cleaned up to correctly stash and then retrieve the
actual error.
Additionally, as I encountered places where error codes were not
being propagated correctly, I tried to fix them up. A number of
those fixes are included in the this commit as well.
This adds giterr_user_cancel to return GIT_EUSER and clear any
error message that is sitting around. As a result of using that
in places, we need to be more thorough with capturing errors that
happen inside a callback when used internally. To help with that,
this also adds giterr_capture and giterr_restore so that when we
internally use a foreach-type function that clears errors and
converts them to GIT_EUSER, it is easier to restore not just the
return value, but the actual error message text.
This adds `git_config__lookup_entry` which will look up a key in
a config and return either the entry or NULL if the key was not
present. Optionally, it can either suppress all errors or can
return them (although not finding the key is not an error for this
function). Unlike other accessors, this does not normalize the
config key string, so it must only be used when the key is known
to be in normalized form (i.e. all lower-case before the first dot
and after the last dot, with no invalid characters).
This also adds three high-level helper functions to look up config
values with no errors and a fallback value. The three functions
are for string, bool, and int values, and will resort to the
fallback value for any error that arises. They are:
* `git_config__get_string_force`
* `git_config__get_bool_force`
* `git_config__get_int_force`
None of them normalize the config `key` either, so they can only
be used for internal cases where the key is known to be in normal
format.
The frontend used to look at the file directly, but that's obviously not
the right thing to do. Expose it on the backend and use that function
instead.
git-core only writes to the reflogs of HEAD, refs/heads/ and,
refs/notes/ or if there is already a reflog in place. Adjust our code to
follow these semantics.
When doing copy detection, it is often necessary to include
UNMODIFIED records in the git_diff so they are available as source
records for GIT_DIFF_FIND_COPIES_FROM_UNMODIFIED. Yet in the final
diff, often you will not want to have these UNMODIFIED records.
This adds a flag which marks these UNMODIFIED records for deletion
from the diff list so they will be removed after the rename detect
phase is over.
When FIND_COPIES is used in combination with BREAK_REWRITES for
rename detection, there was a bug where the split MODIFIED delta
was only used as a target for RENAME records and not for COPIED
records. This fixes that, converting the split into a pair of
DELETED and COPIED deltas when that circumstance arises.
Whenever a reference is created or updated, we need to write to the
reflog regardless of whether the user gave us a message, so we shouldn't
leave that to the ref frontend, but integrate it into the backend.
This also eliminates the race between ref update and writing to the
reflog, as we protect the reflog with the ref lock.
As an additional benefit, this reflog append on the backend happens by
appending to the file instead of parsing and rewriting it.
Copy the pointers into temporary vectors instead of assigning them tot
he same array so we don't mess up with someone else's memory by
accident (e.g. by sorting).
The callback-based method of listing remote references dates back to the
beginning of the network code's lifetime, when we didn't know any
better.
We need to keep the list around for update_tips() after disconnect() so
let's make use of this to simply give the user a pointer to the array so
they can write straightforward code instead of having to go through a
callback.
Removing arbitrary refspecs makes things more complex to reason
about. Instead, let the user set the fetch and push refspec list to
whatever they want it to be.
Create a git_branch_iterator type which is equivalent to the foreach but
lets us write loops instead of callbacks.
Since the introduction of git_reference_shorthand(), the added value of
passing the name is reduced.
When the filesystem iterator encounters an error with a file, it
returns the error but because of the cleanup code, it was in some
cases erasing the error message. This uses the giterr_detach API
to make sure that the actual error message is restored after the
cleanup code has been run.
There are a number of cases where it is convenient to be able to
fetch and "claim" the current error string, clearing the error.
This is helpful when you need to call some code that may alter
the error and you want to restore it later on and/or report it via
some other mechanism.
We used to move `data_start` forward, which is wrong as that needs to
point to the beginning of the buffer in order to perform size
calculations.
Introduce a `write_start` variable which indicates where we should start
writing from, which is what the `data_start` was being wrongly reused to
be.
The last commit taught git_checkout_tree to actually do something
meaningfull, when treeish was NULL. This lets us rewrite
git_checkout_head to simply call git_checkout_tree without giving it a
treeish.
In git_checkout_tree, the first check tests if either repo or treeish is
NULL and says that eithor of them has to have a valid value. But there
is no code to handle the treeish == NULL case.
So, do something meaningful in that case: use HEAD instead.
When downloading the default branch due to lack of refspecs, we still
need to write out FETCH_HEAD with the tip we downloaded, unfortunately
with a format that doesn't match what we already have.
This avoids sending our whole history bit by bit to the remote in cases
where there is no common history, just to give up in the end.
The number comes from the canonical implementation.
The correct behaviour when a remote has no refspecs (e.g. a URL from the
command-line) is to download the remote's HEAD. Let's do that.
This fixes#1261.
This was never really working right because we were checking the
wrong flag and not checking it in all the places that we need to
be checking it. I finally got around to writing a test and adding
actual support for it.
Sometimes the static initializer for git_diff_options cannot be
used and since setting them to all zeroes doesn't actually work
quite right, this adds a new helper for that situation.
This also adds an explicit new value to the submodule settings
options to be used when those enums need static initialization.
This changes `git_index_read` to have two modes - a hard index
reload that always resets the index to match the on-disk data
(which was the old behavior) and a soft index reload that uses
the timestamp / file size information and only replaces the index
data if the file on disk has been modified.
This then updates the git_status code to do a soft reload unless
the new GIT_STATUS_OPT_NO_REFRESH flag is passed in.
This also changes the behavior of the git_diff functions that use
the index so that when an index is not explicitly passed in (i.e.
when the functions call git_repository_index for you), they will
also do a soft reload for you.
This intentionally breaks the file signature of git_index_read
because there has been some confusion about the behavior previously
and it seems like all existing uses of the API should probably be
examined to select the desired behavior.
These changes fix the basic problem with GIT_DIFF_REVERSE being
broken for text diffs. The reversed diff entries were getting
added to the git_diff correctly, but some of the metadata was kept
incorrectly in a way that prevented the text diffs from being
generated correctly. Once I fixed that, it became clear that it
was not possible to merge reversed diffs correctly. This has a
first pass at fixing that problem. We probably need more tests
to make sure that is really fixed thoroughly.
Seems that regexp in Mac OS X and Linux were behaving
differently: while in OS X the empty string didn't
match any value, in Linux it was matching all of them,
so the the second fetch refspec was overwritting the
first one, instead of creating a new one.
Using an unmatcheable regular expression solves the
problem (and seems to be portable).
At some moment git_config_delete_entry lost the ability to delete one entry of
a multivar configuration. The moment you had more than one fetch or push
ref spec for a remote you will not be able to save that remote anymore. The
changes in network::remote::remotes::save show that problem.
I needed to create a new git_config_delete_multivar because I was not able to
remove one or several entries of a multivar config with the current API.
Several tries modifying how git_config_set_multivar(..., NULL) behaved were
not successful.
git_config_delete_multivar is very similar to git_config_set_multivar, and
delegates into config_delete_multivar of config_file. This function search
for the cvar_t that will be deleted, storing them in a temporal array, and
rebuilding the linked list. After calling config_write to delete the entries,
the cvar_t stored in the temporal array are freed.
There is a little fix in config_write, it avoids an infinite loop when using
a regular expression (case for the multivars). This error was found by the
test network::remote::remotes::tagopt.
This tells the server that we speak it, but we don't make use of its
extra information to determine if there's a better place to stop
negotiating.
In a somewhat-related change, reorder the capabilities so we ask for
them in the same order as git does.
Also take this opportunity to factor out a fairly-indented portion of
the negotiation logic.
It was there to keep it apart from the one which read in from a file on
disk. This other indexer does not exist anymore, so there is no need for
anything other than git_indexer to refer to it.
While here, rename _add() function to _append() and _finalize() to
_commit(). The former change is cosmetic, while the latter avoids
talking about "finalizing", which OO languages use to mean something
completely different.
When building libgit2 for ia32 architecture on a x64 machine, including
"config.h" without a "common.h" would result the following error:
C:\Program Files\Microsoft SDKs\Windows\v7.1\include\winbase.h(2288): error C2373: 'InterlockedIncrement' : redefinition; different type modifiers [C:\cygwin\home\zcbenz\codes\git-utils\build\libgit2.vcxproj]
C:\Program Files\Microsoft SDKs\Windows\v7.1\include\winbase.h(2295): error C2373: 'InterlockedDecrement' : redefinition; different type modifiers [C:\cygwin\home\zcbenz\codes\git-utils\build\libgit2.vcxproj]
C:\Program Files\Microsoft SDKs\Windows\v7.1\include\winbase.h(2303): error C2373: 'InterlockedExchange' : redefinition; different type modifiers [C:\cygwin\home\zcbenz\codes\git-utils\build\libgit2.vcxproj]
C:\Program Files\Microsoft SDKs\Windows\v7.1\include\winbase.h(2314): error C2373: 'InterlockedExchangeAdd' : redefinition; different type modifiers [C:\cygwin\home\zcbenz\codes\git-utils\build\libgit2.vcxproj]
The user is unable to derive the number of deltas in the pack, as that
would require them to capture the stats exactly in the moment between
download and final processing, which is abstracted away in the fetch.
Capture these numbers for the user and expose them in the progress
struct. The clone and fetch examples now also present this information
to the user.
The names from libssh2 are somewhat obtuse for us. We can simplify the
usual key/passphrase credential's name, as well as make clearer what the
custom signature function is.
It seems that to implement these options, we just have to pass
the appropriate flags through to the libxdiff code taken from
core git. So let's do it (and add a test).
Instead of having functions with so very many parameters to pass
hunk and line data, this takes the existing git_diff_hunk struct
and extends it with more hunk data, plus adds a git_diff_line.
Those structs are used to pass back hunk and line data instead of
the old APIs that took tons of parameters.
Some work that was previously only being done for git_diff_patch
creation (scanning the diff content for exact line counts) is now
done for all callbacks, but the performance difference should not
be noticable.
While the base git_diff_delta structure always contains two files,
when we introduce conflict data, it will be helpful to have an
indicator when an additional file is involved.
Move conflict handling into two steps: load the conflicts and
then apply the conflicts. This is more compatible with the
existing checkout implementation and makes progress reporting
more sane.
If a D/F conflict or rename 2->1 conflict occurs,
we write the file sides as filename~branchname. If
a file with that name already exists in the working
directory, write as filename~branchname_0 instead.
(Incrementing 0 until a unique filename is found.)
This lays groundwork for separating formatting options from diff
creation options. This groups the formatting flags separately
from the diff list creation flags and reorders the options. This
also tweaks some APIs to further separate code that uses patches
from code that just looks at git_diffs.
This makes no functional change to diff but renames a couple of
the objects and splits the new git_patch (formerly git_diff_patch)
into a new header file.
Don't increase the number of total objects, as it can produce
suprising progress output. The only addition compared to pre-thin is
the addition of local_objects to allow an output similar to git's
"completed with %d local objects".
The iconv init was accidentally clearing the default error state
during reference normalization. This resets so that normalization
errors will be detected correctly.
Before these changes, looking up a reference would return the
same precomposed or decomposed form of the reference name that
was used to look it up, so on MacOS which ignores the difference
between the two, a single reference could be looked up either way
and git_reference_name would return the form of the name that was
used to look it up! This change makes lookup always return the
precomposed name if core.precomposeunicode is set regardless of
which version was used to look it up. The reference iterator was
already returning the precomposed form from earlier work.
This also updates the CMakeLists.txt rules for enabling iconv
usage because the clar tests for this code were actually not being
activated properly with the old version.
Finally, this moves git_repository_reset_filesystem from include/
git2/repository.h to include/git2/sys/repository.h since it is not
really a function that normal library users should have to think
about very often.
This cleans up some additional issues. The main change is that
on a filesystem that doesn't support mode bits, libgit2 will now
create new blobs with GIT_FILEMODE_BLOB always instead of being
at the mercy to the filesystem driver to report executable or not.
This means that if "core.filemode" lies and claims that filemode
is not supported, then we will ignore the executable bit from the
filesystem. Previously we would have allowed it.
This adds an option to the new git_repository_reset_filesystem to
recurse through submodules if desired. There may be other types
of APIs that would like a "recurse submodules" option, but this
one is particularly useful.
This also has a number of cleanups, etc., for related things
including trying to give better error messages when problems come
up from the filesystem. For example, the FAT filesystem driver on
MacOS appears to return errno EINVAL if you attempt to write a
filename with invalid UTF-8 in it. We try to capture that with a
better error message now.
There may be multiple deltas referencing the same base as well as OFS
deltas which rely on a thin delta. Deal with both at the same time by
injecting a single object and going back up to the main
delta-resolving loop.
When a tool needs to recreate the tree object (for example an
interface to another VCS), it needs to use the raw attributes,
forgoing any normalization.
When a repository is transferred from one file system to another,
many of the config settings that represent the properties of the
file system may be wrong. This adds a new public API that will
refresh the config settings of the repository to account for the
change of file system. This doesn't do a full "reinitialize" and
operates on a existing git_repository object refreshing the config
when done.
This commit then makes use of the new API in clar as each test
repository is set up.
This commit also has a number of other clar test fixes where we
were making assumptions about the type of filesystem, either based
on outdated config data or based on the OS instead of the FS.
When given an ODB from which to read objects, the indexer will attempt
to inject the missing bases at the end of the pack and update the
header and trailer to reflect the new contents.
Though unusual, a packfile may contain a delta whose base is a delta
that comes later. In order index such a packfile, we must not give up
on the first failure to resolve a delta, but keep it around.
If there is a pass which makes no progress, this indicates that the
packfile is broken, so fail accordingly.
The repo init code was assuming Windows == no filemode, and
Mac or Windows == no case sensitivity. Those assumptions are not
consistently true depending on the mounted file system. This is a
first step to removing those assumptions. It focuses on the repo
init code and the tests of that code. There are still many other
tests that are broken when those assumptions don't hold true, but
this clears up one area of the code.
Also, this moves the core.precomposeunicode logic to be closer to
the current logic in core Git where it will be set to true on any
filesystem where composed unicode is decomposed when read back.
The indexer code was generating warnings on Windows 64-bit. I
looked closely at the logic and was able to simplify it a bit.
Also this fixes some other Windows and Linux warnings.
This adds a simple wrapper around the iconv APIs and uses it
instead of the old code that was inlining the iconv stuff. This
makes it possible for me to test the iconv logic in isolation.
A "no iconv" version of the API was defined with macros so that
I could have fewer ifdefs in the code itself.
This simplifies git_path_is_empty_dir on both Windows (getting rid
of git_buf allocation inside the function) and other platforms (by
just using git_path_direach), and adds tests for the function, and
uses the function to simplify some existing tests.
This hooks up git_path_direach and git_path_dirload so that they
will take a flag indicating if directory entry names should be
tested and converted from decomposed unicode to precomposed form.
This code will only come into play on the Apple platform and even
then, only when certain types of filesystems are used.
This involved adding a flag to these functions which involved
changing a lot of places in the code.
This was an opportunity to do a bit of code cleanup here and there,
for example, getting rid of the git_futils_cleanupdir_r function in
favor of a simple flag to git_futils_rmdir_r to not remove the top
level entry. That ended up adding depth tracking during rmdir_r
which led to a safety check for infinite directory recursion. Yay.
This hasn't actually been tested on the Mac filesystems where the
issue occurs. I still need to get test environment for that.
This doesn't actual do string precompose but it puts the hooks in
place into the iterators and the git_path_dirload function so that
the actual precompose work is ready to go.
This adds initialization of core.precomposeunicode to repo init
on Mac. This is necessary because when a Mac accesses a repo on
a VFAT or SAMBA file system, it will return directory entries in
decomposed unicode even if the filesystem entry is precomposed.
This also removes caching of a number of repo properties from the
repo init pipeline because these are properties of the specific
filesystem on which the repo is created, not of the system as a
whole.
This commit adds cancellation for the push operation. This work consists of:
1) Support cancellation during push operation
- During object counting phase
- During network transfer phase
- Propagate GIT_EUSER error code out to caller
2) Improve cancellation support during fetch
- Handle cancellation request during network transfer phase
- Clear error string when cancelled during indexing
3) Fix error handling in git_smart__download_pack
Cancellation during push is still only handled in the pack building and
network transfer stages of push (and not during packbuilding).
References and their logs are logically coupled, let's make it so in
the code by moving the fs-based reflog implementation to live next to
the fs-based refs one.
As part of the change, make the function take names rather than
references, as only the names are relevant when looking up and
handling reflogs.
The basic clone function is there to make it easy to create a "normal"
clone. Remove a bunch of options that are about changing the remote's
configuration.
The text progress and update_tips callbacks are already part of the
struct, which was meant to unify the callback setup, but the download
one was left out.
This adds the basics of progress reporting during push. While progress
for all aspects of a push operation are not reported with this change,
it lays the foundation to add these later. Push progress reporting
can be improved in the future - and consumers of the API should
just get more accurate information at that point.
The main areas where this is lacking are:
1) packbuilding progress: does not report progress during deltafication,
as this involves coordinating progress from multiple threads.
2) network progress: reports progress as objects and bytes are going
to be written to the subtransport (instead of as client gets
confirmation that they have been received by the server) and leaves
out some of the bytes that are transfered as part of the push protocol.
Basically, this reports the pack bytes that are written to the
subtransport. It does not report the bytes sent on the wire that
are received by the server. This should be a good estimate of
progress (and an improvement over no progress).
The subtransport path was relying on pointing to data owned by
the remote which meant that after a redirect, the updated path
was getting lost for future requests. This updates the http
transport to strdup the path and maintain its own lifetime.
This also pulls responsibility for parsing the URL back into the
http transport and isolates the functions that parse and free that
connection data so that they can be reused between the initial
parsing and the redirect parsing.
On occasion, files can disappear while we're iterating the
filesystem, between calls to readdir and stat. Let's pretend
those didn't exist in the first place.
The git_buf_text_gather_stats call returns a boolean indicating if
the file looks like binary data. That shouldn't be an error; it
should be used to skip CRLF processing though.
This replaces some git_buf_printf calls with simple calls to
git_buf_put instead. Also, it fixes a missing va_end inside
the git_buf_vprintf implementation.
The attempt to "clean up warnings" seems to have introduced some
new warnings on compliant compilers. This fixes those in a way
that I suspect will also be okay for the non-compliant compilers.
Also this fixes what appears to be an extra semicolon in the
repo initialization template dir handling (and as part of that
fix, handles the case where an error occurs correctly).
In revwalk, we are doing a very simple check to see if a string
contains wildcard characters, so a full regular expression match
is not needed.
In remote listing, now that we have git_config_foreach_match with
full regular expression matching, we can take advantage of that
and eliminate the regex here, replacing it with much simpler string
manipulation.
This contains a few bug fixes and some header and API cleanups.
The main API change is that filters should now use GIT_PASSTHROUGH
to indicate that they wish to skip processing a file instead of
GIT_ENOTFOUND.
The bug fixes include a possible out-of-range buffer access in
the ident filter, a filter ordering problem I introduced into the
custom filter tests on Windows, and a filter buf NUL termination
issue that was coming up on Linux.
This adds more tests of filters, including the ident filter when
mixed with custom filters. I was able to combine with the reverse
filter and demonstrate that the order of filter application with
the default priority constants matches the order of core Git.
Also, this fixes two issues in the ident filter: preventing ident
expansion on binary files and avoiding a NULL dereference when
dollar sign characters are found without Id.
Fixed the filter order to match core Git, too.
This test demonstrates an interesting behavior of core Git (which
is totally reasonable and which libgit2 matches, although mostly
by coincidence). If you use the ident filter and commit a file
with a garbage ident in it, like '$Id: this is just garbage$' and
then immediately do a 'git checkout-index' with the new file, Git
will not consider the file out of date and will not overwrite the
file with an updated $Id$. Libgit2 has the same behavior. If you
remove the file and then do a checkout-index, it will be replaced
with a filtered version that has injected the OID correctly.
These are a couple of new clar helpers for testing that a file
has expected contents that I extracted from the checkout code.
Actually wrote this as part of an abandoned earlier attempt at a
new filters API, but it will be useful now for some of the tests
I'm going to write.
I wish MSVC understood that "const char **" is not a const ptr,
but it a non-const pointer to an array of const ptrs. Does that
seem like too much to ask.
Checkout should not reject binary files from filters, as a filter
may actually wish to operate on binary files. The CRLF filter should
reject binary files itself if it wishes to. Moreover, the CRLF
filter requires this logic so that users can emulate the checkout
data in their odb -> workdir filtering.
Conflicts:
src/checkout.c
src/crlf.c
This makes the git_buf struct that was used internally into an
externally available structure and eliminates the git_buffer.
As part of that, some of the special cases that arose with the
externally used git_buffer were blended into the git_buf, such as
being careful about git_buf objects that may have a NULL ptr and
allowing for bufs with a valid ptr and size but zero asize as a
way of referring to externally owned data.
This adds the ident filter (that knows how to replace $Id$) and
tweaks the filter APIs and code so that git_filter_source objects
actually have the updated OID of the object being filtered when
it is a known value.
Extend the git2/sys/filter API with functions to look up a filter
and add it manually to a filter list. This requires some trickery
because the regular attribute lookups and checks are bypassed when
this happens, but in the right hands, it will allow a user to have
granular control over applying filters.
Increasingly there are a number of components that want to do some
cleanup at global shutdown time (at least if there are not going
to be memory leaks). This creates a very simple system of shutdown
hooks that will be invoked by git_threads_shutdown. Right now, the
maximum number of hooks is hardcoded, but since adding a hook is
not a public API, it should be fine and I thought it was better to
start off with really simple code.
This moves the git_filter_list into the public API so that users
can create, apply, and dispose of filter lists. This allows more
granular application of filters to user data outside of libgit2
internals.
This also converts all the internal usage of filters to the public
APIs along with a few small tweaks to make it easier to use the
public git_buffer stuff alongside the internal git_buf.
The filter registry as implemented was too primitive to actually
work once multiple filters were coming into play. This expands
the implementation of the registry to handle multiple prioritized
filters correctly.
Additionally, this adds an "attributes" field to a filter that
makes it really really easy to implement filters that are based
on one or more attribute values. The lookup and even simple value
checking can all happen automatically without custom filter code.
Lastly, with the registry improvements, this fills out the filter
lifecycle callbacks, with initialize and shutdown callbacks that
will be called before the filter is first used and after it is
last invoked. This allows for system-wide initialization and
cleanup by the filter.
This creates include/sys/filter.h with a basic definition of a
git_filter and then converts the internal code to use it. There
are related internal objects (git_filter_list) that we will want
to publish at some point, but this is a first step.
This begins the process of exposing git_filter objects to the
public API. This includes:
* new public type and API for `git_buffer` through which an
allocated buffer can be passed to the user
* new API `git_blob_filtered_content`
* make the git_filter type and GIT_FILTER_TO_... constants public
Unfortunately git-core uses the term "unborn branch" and "orphan
branch" interchangeably. However, "orphan" is only really there for
the checkout command, which has the `--orphan` option so it doesn't
actually create the branch.
Branches never have parents, so the distinction of a branch with no
parents is odd to begin with. Crucially, the error messages deal with
unborn branches, so let's use that.
As the include depth increases, the chance of a realloc
increases. This means that whenever we run git_array_alloc() or call
config_parse(), we need to remember what our reader's index is so we
can look it up again.
We need to refresh the variables from the included files if they are
changed, so loop over all included files and re-parse the files if any
of them has changed.
Now that #1785 is merged, git_odb_stream_finalize_write() calculates the object id before invoking the odb backend.
This commit gives a chance to the backend to check if it already knows this object.
The GIT_MODE_TYPE macro was looking at all bits above the
permissions, but it should really just look at the top bits so
that it will give the right results for a setgid or setuid entry.
Since we're now using these macros in the tests, this was causing
a test failure on platforms that don't support setgid.
This adds some more macros for some standard operations on file
modes, particularly related to permissions, and then updates a
number of places around the code base to use the new macros.
In order to support config includes, we must differentiate between the
backend's main file and the file we are currently parsing.
This lays the groundwork for includes, keeping the current behaviours.
Previously, `git_object_read()`, `git_object_read_prefix()` and
`git_object_exists()` were implementing an auto refresh logic. When the
expected object couldn't be found in any backend, a call to
`git_odb_refresh()` was triggered and the lookup was once again performed
against all backends.
This commit removes this auto-refresh logic from the odb layer and pushes
it down into the pack-backend (as it's the only one currently exposing
a `refresh()` endpoint).
This simplifies the git_repository_is_empty a bit so that a
detached HEAD is just taken to mean the repo is not empty, since
a newly initialized repo will not have a detached HEAD.
Ensure that we apply splits to rewrites, even if we're not
interested in examining it closely for rename/copy detection.
In keeping with core git, status should not display rewrites,
it should simply show files as "modified".
In order to be loaded, a remote needs to be configured with at least a `url` or a `pushurl`.
ENOTFOUND will be returned when trying to git_remote_load() a remote with neither of these entries defined.
This loads SRWLock APIs at runtime and in their absence (i.e. on
Windows before Vista) falls back on a regular CRITICAL_SECTION
that will not permit concurrent readers.
9e9aee6 added an include <netinet/in.h> to fix the build on FreeBSD.
Sometime since then the same header is included ifndef _WIN32, so
remove the duplicate include.
This converts an internal lock from a write lock to a read lock
where write isn't needed, and also clarifies some doc things about
where various locks are acquired and how various APIs are intended
to be used.
This adds thread safety to the refdb_fs by using the new
git_sortedcache object and also by relaxing the handling of some
filesystem errors where the fs may be changed out from under us.
This also adds some new threading tests that hammer on the refdb.
The refdb_fs implementation calls realloc directly on a reference
object when it wants to rename it. It is not a public object, so
this doesn't mess with the immutability of references, but it does
assume certain constraints on the reference representation. This
commit wraps that assumption in an isolated API to isolate it.
This adds a convenient new data type for caching the contents of
file in memory when each item in that file corresponds to a name
and you need to both be able to lookup items by name and iterate
over them in some sorted order. The new data type has locks in
place to manage usage in a threaded environment.
If there were symbolic refs among the loose refs then the code
to create packed-refs would fail trying to parse the OID out of
them (where Git just skips trying to pack them). This fixes it.
When a git_buf contains a UTF-8 BOM, the three bytes comprising
that BOM are treated as unprintable characters. For a small git_buf,
the three BOM characters overwhelm the printable characters. This
is problematic when trying to check out a small file as the CR/LF
filtering will not apply.
p_inet_pton on Windows should set errno properly for callers.
Rewrite p_inet_pton to handle error cases correctly and add
test cases to exercise this function.
Report the index being locked with its own error code in order to be
able to differentiate, as a locked index is typically the result of a
crashed process or concurrent access, both of which often require user
intervention to fix.
If none of the backends support direct writes and we must stream the
whole file, we already know what the object's id should be; so use the
stream's functions directly, bypassing the frontend's hashing and
overwriting of our existing id.
The frontend is in charge of calculating the id of the objects. Thus
the backends should treat it as a read-only value. The positioning in
the function signature made it seem as though it was an output
parameter.
Make the id const and move it from the front to behind the subject
(backend or stream).
When dealing with a chain of tags, we need to enqueue each of them
individually, which means we can't use `git_tag_peel` as that jumps
over the intermediate tags.
Do the peeling manually so we can look at each object and take the
appropriate action.
Hash the data as it's coming into the stream and tell the backend what
its name is when finalizing the write. This makes it consistent with
the way a plain git_odb_write() performs the write.
This is in preparation for moving the hashing to the frontend, which
requires us to handle the incoming data before passing it to the
backend's stream.
This fixes a small memory leak in git_revparse where early returns on
errors from git_revparse_single cause a free() on the (reallocated) left
side of the revspec to be skipped.
Accept any value for the remote's url, including an empty string which
we used to reject as invalid configuration.
This is not quite what git does (although it has its own problems with
such configurations) and it makes it harder to fix the issue, by not
letting the user modify it.
As we already need to check for a valid URL when we try to connect to
the network, let that perform the check, as we don't need to do it
anywhere else.
This reverts refactoring done in 13224ea4aa
that introduces a performance regression for NFS when reading files that
don't exist. open() forces a cache invalidation on NFS, while stat()ing a
file just uses the cache and is very quick.
To give a specific example, say you have a repo with a thousand packed
refs. Before this change, looking up every single one ould incur a thousand
slow open() calls. With this change, it's a thousand fast stat() calls.
This is just a bunch of small fixes that I noticed while looking
at the UTF8 and UTF16 path stuff. It fixes a slowdown in looking
for an empty directory (not exiting loop asap), makes the dir name
in the git__DIR structure be a GIT_FLEX_ARRAY to save an allocation,
and fixes some slightly odd assumptions in the cl_getenv helper.
Key-based authentication also needs an username, so include it in each
one.
Also stop assuming a default username of "git" in the ssh transport
which has no business making such a decision.
The routines to push and pop ignore files while traversing a
directory had some issues. In particular, setting up the initial
list would sometimes push an ignore file before it ought to be
applied if the starting path was a directory containing an ignore
file. Also, the pop function was not always matching the right
part of the path and would fail to pop ignores from the list in
some cases.
This adds some tests that exercise a particular problematic case
and then fixes the problems that I could find related to this.
At some point, I'd like to isolate this ignore rule management
code and rewrite it, but that's a larger project and right now,
I'll opt to just try to fix the broken behaviors.
This rolls back the changes to fnmatch parsing from commit
2e40a60e84 except for the tests
that were added. Instead this adds couple of new flags that can
be passed in when attempting to parse an fnmatch pattern. Also,
this changes the pathspec match logic to special case matching a
filename with a '!' prefix against a negative pattern.
This fixes the build.
`git_config_set_string(config, "config.section", "")` fails when
escaping the value.
The buffer in `escape_value` is allocated without NULL-termination. And
in case of empty string 0 is passed for buffer size in `git_buf_grow`.
`git_buf_detach` returns NULL when the allocated size is 0 and that
leads to an error return in `GITERR_CHECK_ALLOC` called after
`escape_value`
The change in `config_file.c` was suggested by Russell Belfer <rb@github.com>
new functions in struct git_config_backend:
* iterator_new(...)
* iterator_free(...)
* next(...)
The old callback based foreach style can still be used with `git_config_backend_foreach_match`
This step is needed to easily add iterators to git_config_backend
As well use these new git_strmap functions to implement foreach
* git_strmap_iter
* git_strmap_has_data(...)
* git_strmap_begin(...)
* git_strmap_end(...)
* git_strmap_next(...)
In git_diff_paired_foreach, temporarily resort the
index->workdir diff list by index path so that we can
track a rename in the workdir from head->index->workdir.
When using a rename source that is actually a to-be-split record,
we have to update the best-fit mapping data in both the case where
the target is also a split record and the case where the target
is a simple added record. Before this commit, we were only doing
the update when the target was itself a split record (and even in
that case, the test was slightly wrong).
After doing further profiling, I found that a lot of time was
being spent attempting to insert hashes into the file hash
signature when using the rolling hash because the rolling hash
approach generates a hash per byte of the file instead of one
per run/line of data.
To optimize this, I decided to convert back to a run-based file
signature algorithm which would be more like core Git.
After changing this, a number of the existing tests started to
fail. In some cases, this appears to have been because the test
was coded to be too specific to the particular results of the file
similarity metric and in some cases there appear to have been bugs
in the core rename detection code where only by the coincidence
of the file similarity scoring were the expected results being
generated.
This renames all the variables in the core rename detection code
to be more consistent and hopefully easier to follow which made it
a bit easier to reason about the behavior of that code and fix the
problems that I was seeing. I think it's in better shape now.
There are a couple of tests now that attempt to stress test the
rename detection code and they are quite slow. Most of the time
is spent setting up the test data on disk and in the index. When
we roll out performance improvements for index insertion, it
should also speed up these tests I hope.
The size data in the index may not reflect the actual size of the
blob data from the ODB when content filtering comes into play.
This commit fixes rename detection to use the actual blob size when
calculating data signatures instead of the value from the index.
Because of a misunderstanding on my part, I first converted the
git_index_add_bypath API to use the post-filtered blob data size
in creating the index entry. I backed that change out, but I
kept the overall refactoring of that routine and the new internal
git_blob__create_from_paths API because it eliminates an extra
stat() call from the code that adds a file to the index.
The existing tests actually cover this code path, at least when
running on Windows, so at this point I'm not adding new tests to
cover the changes.
The previous fix for checking file sizes with rename detection
always loads the blob. In this version, if the odb backend can
get the object header without loading the whole thing into memory,
then we'll just use that, so that we can eliminate possible rename
sources & targets without loading them.
The performance improvements I introduced for rename detection
were not able to run successfully for tree-to-tree diffs because
the blob size was not known early enough and so the file signature
always had to be calculated nonetheless.
This change separates loading blobs into memory from calculating
the signature. I can't avoid having to load the large blobs into
memory, but by moving it forward, I'm able to avoid the signature
calculation if the blob won't come into play for renames.
This restores the usage of GIT_DIFF_LINE_BINARY for the diff
output line that reads "Binary files x and y differ" so that it
can be optionally colorized independently of the file header.
This allows git_diff_patch_size to account for hunk headers and
file headers in the returned size. This required some refactoring
of the code that is used to print file headers so that it could be
invoked by the git_diff_patch_size API.
Also this increases the test coverage and fixes an off-by-one bug
in the size calculation when newline changes happen at the end of
the file.
Instead of using lots of strdup calls, this adds a memory pool to
the loose refs iteration code and uses it for keeping track of the
loose refs array. Memory usage could probably be reduced even
further by eliminating the vector and just scanning by adding the
strlen of each ref, but that would be a more intrusive changes.
This also updates the error handling to be more thorough about
checking for failed allocations, etc.
The git_reference_next API silently skips invalid references when
scanning the loose refs. The git_reference_next_name API should
skip the same ones even though it isn't creating the reference
object.
This adds a test with a an invalid loose reference and makes sure
that both APIs skip the same entries and generate the same results.
This makes git__swap use the __sync_lock_test_and_set primitive
with GCC and the InterlockedExchangePointer primitive with MSVC.
Previously is used compare_and_swap in a way that was probably
unintuitive for most thinking (i.e. it could fail to swap in the
value if another thread raced in). Now it will always succeed
and the last thread to run in a race will win instead of the
first thread.
This also fixes up a little confusion between volatile void **
and void * volatile * that came up with the Win32 compiler.
This restores a behavior that was accidentally lost during some
diff refactoring where an untracked directory that contains a .git
item should be treated as IGNORED, not as UNTRACKED. The submodule
code already detects this, but the diff code was not handling the
scenario right.
This also updates a number of existing tests that were actually
exercising the behavior but did not have the right expectations in
place. It actually makes the new
`test_diff_submodules__diff_ignore_options` test feel much better
because the "not-a-submodule" entries are now ignored instead of
showing up as untracked items.
Fixes#1697
This adds correct support for an equivalent to --ignore-submodules
in diff, where an actual ignore value can be passed to diff to
override the per submodule settings in the configuration.
This required tweaking the constants for ignore values so that
zero would not be used and could represent an unset option to the
diff. This was an opportunity to move the submodule values into
include/git2/types.h and to rename the poorly named DEFAULT values
for ignore and update constants to RESET instead.
Now the GIT_DIFF_IGNORE_SUBMODULES flag is exactly the same as
setting the ignore_submodules option to GIT_SUBMODULE_IGNORE_ALL
(which is actually a minor change from the old behavior in that
submodules will now be treated as UNMODIFIED deltas instead of
being left out totally - if you set GIT_DIFF_INCLUDE_UNMODIFIED).
This includes tests for the various new settings.
Submodules now expose an internal status API that allows diff to
get back the OID values from the submodule very easily and also
to avoiding caching issues and to override the ignore setting for
the submodule.
This fixes the way that submodule status is checked to bypass just
about all of the caching in the submodule object. Based on the
ignore value, it will try to do the minimum work necessary to find
the current status of the submodule - but it will actually go to
disk to get all of the current values.
This also removes the custom refcounting stuff in favor of the
common git_refcount style. Right now, it is still for internal
purposes only, but it should make it easier to add true submodule
refcounting in the future with a public git_submodule_free call
that will allow bindings not to worry about the submodule object
getting freed from underneath them.
This adds a BARE option to git_repository_open_ext which allows
a fast open path that still knows how to read gitlinks and to
search for the actual .git directory from a subdirectory.
`git_repository_open_bare` is still simpler and faster, but having
a gitlink aware fast open is very useful for submodules where we
want to quickly be able to peek at the HEAD and index data without
doing any other meaningful repo operations.
This is probably not the final form of this change, but this is
a preliminary version of checking a timestamp to see if the cached
working directory HEAD OID matches the current. Right now, this
uses the timestamp on the index and is, like most of our timestamp
checking, subject to having only second accuracy.
This adds an additional pathspec API that will match a pathspec
against a diff object. This is convenient if you want to handle
renames (so you need the whole diff and can't use the pathspec
constraint built into the diff API) but still want to tell if the
diff had any files that matched the pathspec.
When the pathspec is matched against a diff, instead of keeping
a list of filenames that matched, instead the API keeps the list
of git_diff_deltas that matched and they can be retrieved via a
new API git_pathspec_match_list_diff_entry.
There are a couple of other minor API extensions here that were
mostly for the sake of convenience and to reduce dependencies
on knowing the internal data structure between files inside the
library.
This is a simple bit vector object that is not resizable after
the initial allocation but can be of arbitrary size. It will
keep the bti vector entirely on the stack for vectors 64 bits
or less, and will allocate the vector on the heap for larger
sizes. The API is uniform regardless of storage location.
This is very basic right now and all the APIs are inline functions,
but it is useful for storing an array of boolean values.
This converts the array of parent SHAs from a git_vector where
each SHA has to be separately allocated to a git_array_t where
all the SHAs can be kept in one block. Since the two collections
have almost identical APIs, there isn't much involved in making
the change. I did add an API to git_array_t so that it could be
allocated at a precise initial size.
This fixes the way the example log program decides if a merge
commit should be shown when a pathspec is given. Also makes it
easier to use the pathspec API to just check "does a tree match
anything in the pathspec" without allocating a match list.
This adds a new public API for compiling pathspecs and matching
them against the working directory, the index, or a tree from the
repository. This also reworks the pathspec internals to allow the
sharing of code between the existing internal usage of pathspec
matching and the new external API.
While this is working and the new API is ready for discussion, I
think there is still an incorrect behavior in which patterns are
always matched against the full path of an entry without taking
the subdirectories into account (so "s*" will match "subdir/file"
even though it wouldn't with core Git). Further enhancements are
coming, but this was a good place to take a functional snapshot.
The SSH error checking and reporting could still be further
improved by using the libssh2 native methods to get error info,
but at least this ensures that all error codes are checked and
translated into libgit2 error messages.
If there is not an error, the return value was always the return value
of the last call to file->get_multivar
With this commit GIT_ENOTFOUND is only returned if all the calls to
filge-get_multivar return GIT_ENOTFOUND.
This makes all of the credential objects use the same pattern to
clear the contents and call git__memzero when done. Much of this
information is probably not sensitive, but it also seems better
to just clear consistently.
Much of the SSH credential creation API can be left enabled even
on platforms with no SSH support. We really just have to give an
error when you attempt to open the SSH connection.
The diff hunk context string that is returned to xdiff need not
be NUL terminated because the xdiff code just copies the number of
bytes that you report directly into the output. There was an off
by one in the diff driver code when the header context was longer
than the output buffer size, the output buffer length included
the NUL byte which was copied into the hunk header.
Fixes#1710
This option serves no benefit now that the git_status_list API
is available. It was of questionable value before and now it
would just be a bad idea to use it rather than the indexed API.
The index isn't really thread safe for the most part, but we can
easily be more careful and avoid double frees and the like, which
are serious problems (as opposed to a lookup which might return
the incorrect value but if the index in being updated, that is
much harder to avoid).
In both of these cases, the submodule data should still be loaded
just (obviously) without the data that comes from either the index
or the HEAD.
This fixes a bug in the orphaned head case.
There was a bug where submodules whose HEAD had not been moved
were being marked as having an UNMODIFIED delta record instead
of being left MODIFIED. This fixes that and fixes the tests to
notice if a submodule has been incorrectly marked as UNMODIFIED.
In theory, p_stat should never return an S_ISLNK result, but due
to the current implementation on Windows with mount points it is
possible that it will. For now, work around that by allowing a
link in the path to a directory being created. If it is really a
problem, then the issue will be caught on the next iteration of
the loop, but typically this will be the right thing to do.
This updates the calls that make the subdirectories for objects
to use a base directory above which git_futils_mkdir won't walk
any higher. This prevents attempts to mkdir all the way up to
the root of the filesystem.
Also, this moves the objects_dir into the loose backend structure
and removes the separate allocation, plus does some preformatting
of the objects_dir value to guarantee a trailing slash, etc.
With the new target directory option to checkout, the non-bareness
of the repository should be checked much later in the parameter
validation process - actually that check was already in place, but
I was doing it redundantly in the checkout APIs.
This removes the now unnecessary early check for bare repos. It
also adds some other parameter validation and makes it so that
implied parameters can actually be passed as NULL (i.e. if you
pass a git_index, you don't have to pass the git_repository - we
can get it from index).
This adds the ability for checkout to write to a target directory
instead of having to use the working directory of the repository.
This makes it easier to do exports of repository data and the like.
This is similar to, but not quite the same as, the --prefix option
to `git checkout-index` (this will always be treated as a directory
name, not just as a simple text prefix).
As part of this, the workdir iterator was extended to take the
path to the working directory as a parameter and fallback on the
git_repository_workdir result only if it's not specified.
Fixes#1332
This fixes the checkout case when a file is modified between the
baseline and the target and yet missing in the working directory.
The logic for that case appears to have been wrong.
This also adds a useful checkout notify callback to the checkout
test helpers that will count notifications and also has a debug
mode to visualize what checkout thinks that it's doing.
Files in status will, be default, be sorted according to the case
insensitivity of the filesystem that we're running on. However,
in some cases, this is not desirable. Even on case insensitive
file systems, 'git status' at the command line will generally use
a case sensitive sort (like 'ls'). Some GUIs prefer to display a
list of file case insensitively even on case-sensitive platforms.
This adds two new flags: GIT_STATUS_OPT_SORT_CASE_SENSITIVELY
and GIT_STATUS_OPT_SORT_CASE_INSENSITIVELY that will override the
default sort order of the status output and give the user control.
This includes tests for exercising these new options and makes
the examples/status.c program emulate core Git and always use a
case sensitive sort.
This adds some tests for updating the index and having it remove
items to make sure that the iteration over the index still works
even as earlier items are removed.
In testing with valgrind, this found a path that would use the
path string from the index entry after it had been freed. The
bug fix is simply to copy the path of the index entry before
doing any actual index manipulation.
This adds three new public APIs for manipulating the index:
1. `git_index_add_all` is similar to `git add -A` and will add
files in the working directory that match a pathspec to the
index while honoring ignores, etc.
2. `git_index_remove_all` removes files from the index that match
a pathspec.
3. `git_index_update_all` updates entries in the index based on
the current contents of the working directory, either added
the new information or removing the entry from the index.
Command line Git sometimes generates an error message if given a
pathspec that contains an exact match to an ignored file (provided
--force isn't also given). This adds an internal function that
makes it easy to check it that has happened. Right now, I'm not
creating a public API for this because that would get a little
more complicated with a need for callbacks for all invalid paths.
Right now, setting up a pathspec to be parsed and processed
requires several data structures and a couple of API calls. This
adds a new high level data structure that contains all the items
that you'll need and high-level APIs that do all of the setup and
all of the teardown. This will make it easier to use pathspecs
in various places with less repeated code.
This makes the diff rename tracking code more careful about the
order in which it processes renames and more thorough in updating
the mapping of correct renames when an earlier rename update
alters the index of a later matched pair.
This adds parameters to the four functions that allow for blob-to-
blob and blob-to-buffer differencing (either via callbacks or by
making a git_diff_patch object). These parameters let you say
that filename we should pretend the blob has while doing the diff.
If you pass NULL, there should be no change from the existing
behavior, which is to skip using attributes for file type checks
and just look at content. With the parameters, you can plug into
the new diff driver functionality and get binary or non-binary
behavior, plus function context regular expressions, etc.
This commit also fixes things so that the git_diff_delta that is
generated by these functions will actually be populated with the
data that we know about the blobs (or buffers) so you can use it
appropriately. It also fixes a bug in generating patches from
the git_diff_patch objects created via these functions.
Lastly, there is one other behavior change that may matter. If
there is no difference between the two blobs, these functions no
longer generate any diff callbacks / patches unless you have
passed in GIT_DIFF_INCLUDE_UNMODIFIED. This is pretty natural,
but could potentially change the behavior of existing usage.
This changes the behavior of the status RENAMED flags so that they
will be combined with the MODIFIED flags if appropriate. If a file
is modified in the index and also renamed, then the status code
will have both the GIT_STATUS_INDEX_MODIFIED and INDEX_RENAMED bits
set. If it is renamed but the OID has not changed, then just the
GIT_STATUS_INDEX_RENAMED bit will be set. Similarly, the flags
GIT_STATUS_WT_MODIFIED and GIT_STATUS_WT_RENAMED can both be set
independently of one another.
This fixes a serious bug where the check for unmodified files that
was done at data load time could end up erasing the RENAMED state
of a file that was renamed with no changes.
Lastly, this contains a bunch of new tests for status with renames,
including tests where the only rename changes are case changes.
The expected results of these tests have to vary by whether the
platform uses a case sensitive filesystem or not, so the expected
data covers those platform differences separately.
Trees are always case sensitive. The index is always case
preserving and will be case sensitive when it is turned into a
tree. Therefore the tree and the index can and should always
be compared to one another case sensitively.
This will restore the index to case insensitive order after the
diff has been generated.
Consider this a short-term fix. The long term fix is to have the
index always stored both case sensitively and case insensitively
(at least on platforms that sometimes require case insensitivity).
In a case insensitive index, if you attempt to add a file from
disk with a different case pattern, the old case pattern in the
index should be preserved.
This fixes that (and a couple of minor warnings).
This commit reinstates some changes to git_diff__paired_foreach
that were discarded during the rebase (because the diff_output.c
file had gone away), and also adjusts the case insensitively
logic slightly to hopefully deal with either mismatched icase
diffs and other case insensitivity scenarios.
This makes diff more careful about picking the canonical path
when generating a delta so that it won't accidentally pick up a
case-mismatched path on a case-insensitive file system. This
should make sure we use the "most accurate" case correct version
of the path (i.e. from the tree if possible, or the index if
need be).
The exclude submodules flag was not doing the right thing, in
that a file with no diff between the head and the index and just
a delete in the workdir could be excluded if submodules were
excluded.
On Linux: fix a warning message related to the volatile qualifier (cast)
On Windows: use SecureZeroMemory()
On both, inline the call, so that no entry point can lead back to this "secure" memory zeroing.
This makes the git_diff_patch definition private to diff_patch.c
and fixes a number of other header file naming inconsistencies to
use `git_` prefixes on functions and structures that are shared
between files.
This changes the size data to uint32_t, fixes the array growth
logic to use a simple 1.5x multiplier, and uses a generic inline
function for growing the array to make the git_array_alloc API
feel more natural (i.e. it returns a pointer to the new item).
This adds two new public APIs: git_diff_patch_from_blobs and
git_diff_patch_from_blob_and_buffer, plus it refactors the code
for git_diff_blobs and git_diff_blob_to_buffer so that they code
is almost entirely shared between these APIs, and adds tests for
the new APIs.
This implements the loading of regular expression pattern lists
for diff drivers that search for function context in that way.
This also changes the way that diff drivers update options and
interface with xdiff APIs to make them a little more flexible.
There are all sorts of misconfiguration in the wild. We already rely
on the signature constructor to trim SP. Extend the logic to use
`isspace` to decide whether a character should be trimmed.
This is a significant reorganization of the diff code to break it
into a set of more clearly distinct files and to document the new
organization. Hopefully this will make the diff code easier to
understand and to extend.
This adds a new `git_diff_driver` object that looks of diff driver
information from the attributes and the config so that things like
function content in diff headers can be provided. The full driver
spec is not implemented in the commit - this is focused on the
reorganization of the code and putting the driver hooks in place.
This also removes a few #includes from src/repository.h that were
overbroad, but as a result required extra #includes in a variety
of places since including src/repository.h no longer results in
pulling in the whole world.
There are two places where git_futils_mkdir should exit early or
at least do less. The first is when using GIT_MKDIR_SKIP_LAST
and having that flag leave no directory left to create; it was
being handled previously, but the behavior was subtle. Now I put
in a clear explicit check that exits early in that case.
The second is when there is no directory to create, but there is
a valid path that should be verified. I shifted the logic a bit
so we'll be better about not entering the loop than that happens.
This implements a basic callback to extract function context for
a diff. It always uses the same search heuristic right now with
no regular expressions or language-specific variants. Those will
come next, I think.
This makes sure that git_futils_mkdir always skips over the root
directory at a minimum, even on platforms where the root is not
simply '/'. Also, this removes the GIT_WIN32 ifdef in favor of
making EACCES as a potentially recoverable error on all platforms.
We ran into an issue where cloning a repository to a folder
directly underneath the root of a volume (e.g. 'd:\libgit2')
would fail with an access denied error. This was traced down
to a call to make a directory that is the root (e.g. 'd:') could
return an error indicated access denied instead of an error
indicating the path already exists. This change now handles
the access denied error on Win32 and checks for the existence
of the folder.
git doesn't do that, and it's not something that's usually
actionable to fix. if you have a git repository with one bad
timezone in the history, it's too late to change it most likely.
Instead of just blowing away the stat cache data when loading a
new tree into the index, this checks if each loaded item has a
corresponding existing item with the same OID and if so, copies
the stat data from the old item to the new one so it will not be
blown away.
It is obviously quite a serious problem if this happens, but mutex
initialization can fail and we should detect it. It's a bit like
a memory allocation failure, in that you're probably pretty screwed
if this occurs, but at least we'll catch it.
By zeroing out the memory when we free larger objects (i.e. those
that serve as collections of other data, such as repos, odb, refdb),
I'm hoping that it will be easier for libgit2 bindings to find
errors in their object management code.
1. internal iterators now return GIT_ITEROVER when you go past the
last item in the iteration.
2. git_iterator_advance will "advance" to the first item in the
iteration if it is called immediately after creating the
iterator, which allows a simpler idiom for basic iteration.
3. if git_iterator_advance encounters an error reading data (e.g.
a missing tree or an unreadable file), it returns the error
but also attempts to advance past the invalid data to prevent
an infinite loop.
Updated all tests and internal usage of iterators to account for
these new behaviors.
`lpExitCode` is a pointer to a long. A long is 32 bits wide on Windows.
It means that on Windows 64bits, `GetExitCodeThread()` doesn't set/clear the high-order bytes of the 64 bits memory space pointed at by `value_ptr`.
git_packbuilder_write() used to write a packfile to the passed file
path. Instead, ask for a destination directory and create both the
packfile and an index, as most users probably do expect.
This adds ~/ prefix expansion for the value of core.attributesfile
and core.excludesfile, plus it fixes the fact that the attributes
cache was holding on to the string data from the config for a long
time (instead of making its own strdup) which could have caused a
problem if the config was refreshed. Adds a test for the new
expansion capability.
This improves the docs for GIT_DIFF_INCLUDE_UNTRACKED_CONTENT as
well as the other flags related to UNTRACKED items in diff, plus
it makes that flag now automatically turn on
GIT_DIFF_INCLUDE_UNTRACKED which seems like a reasonable dwim type
of change.
The GIT_CONFIG_LEVEL constants actually work well as an enum
because they are mutually exclusive, so this adds a typedef to
the enum and uses that everywhere that one of these constants are
expected, instead of the old code that typically used an unsigned
int.
This adds docs for the cache control options to git_libgit2_opts
and also tweaks the cache code so that if the cache is disabled,
then the next time we attempt to insert something into the cache
in question, we will actually clear any old cached objects.
In theory, if there was a problem reading the REUC data, the
read_reuc() routine could have left uninitialized and invalid
data in the git_index vector. This moves the line that inserts a
new entry into the vector down to the bottom of the routine so we
know all the content is already valid. Also, per @linquize, this
uses calloc to ensure no uninitialized data.
This extends the rename tests to make sure that every rename
scenario in the inner loop of git_diff_find_similar is actually
exercised. Also, fixes an incorrect assert that was in one of
the clauses that was not previously being exercised.
This moves the GIT_CVAR_ABBREV lookup out of the loop. Also, this
fixes git_diff_print_raw to actually use that constant instead of
hardcoding 7 characters.
This adds a couple more tests of different rename scenarios.
Also, this fixes a problem with the case where you have two
"split" deltas and the left half of one matches the right half of
the other. That case was already being handled, but in the wrong
order in a way that could result in bad output. Also, if the swap
also happened to put the other two halves into the correct place
(i.e. two files exchanged places with each other), then the second
delta was left with the SPLIT flag set when it really should be
cleared.
This flips rename detection around so instead of creating a
forward mapping from deltas to possible rename targets, instead
it creates a reverse mapping, looking at possible targets and
trying to find a source that they could have been renamed or
copied from. This is important because each output can only
have a single source, but a given source could map to multiple
outputs (in the form of COPIED records).
Additionally, this makes a couple of tweaks to the public rename
detection APIs, mostly renaming a couple of options that control
the behavior to make more sense and to be more like core Git.
I walked through the tests looking at the exact results and
updated the expectations based on what I saw. The new code is
different from the old because it cannot give some nonsense
results (like A was renamed to both B and C) which were part of
the outputs previously.
This adds a bunch more rename detection tests including checks
vs the working directory, the new exact match options, some more
whitespace variants, etc.
This also adds a git_futils_writebuffer helper function and uses
it in checkout. This is mainly added because I wanted an easy
way to write out a git_buf to disk inside my test code.
- Add new GIT_DIFF_FIND_EXACT_MATCH_ONLY flag to do similarity
matching without using the similarity metric (i.e. only compare
the SHA).
- Clean up the similarity measurement code to more rigorously
distinguish between files that are not similar and files that
are not comparable (previously, a 0 could either mean that the
files could not be compared or that they were totally different)
- When splitting a MODIFIED file into a DELETE/ADD pair, actually
make a DELETED/UNTRACKED pair if the right side of the diff is
from the working directory. This prevents an odd mix of ADDED
and UNTRACKED files on workdir diffs.
There are a number of bugs in the rename code that only were
obvious when I started testing it against large old repos with
more complex patterns. (The code to do that testing is not ready
to merge with libgit2, but I do plan to add more thorough tests.)
This contains a significant number of changes and also tweaks the
public API slightly to make emulating core git easier.
Most notably, this separates the GIT_DIFF_FIND_AND_BREAK_REWRITES
flag into FIND_REWRITES (which adds a self-similarity score to
every modified file) and BREAK_REWRITES (which splits the modified
deltas into add/remove pairs in the diff list). When you do a raw
output of core git, rewrites show up as M090 or such, not at A and
D output, so I wanted to be able to emulate that.
Publicly, this also changes the flags to be uint16_t since we
don't need values out of that range.
Internally, this contains significant changes from a number of
small bug fixes (like using the wrong side of the diff to decide
if the object could be found in the ODB vs the workdir) to larger
issues about which files can and should be compared and how the
various edge cases of similarity scores should be treated.
Honestly, I don't think this is the last update that will have to
be made to this code, but I think this moves us closer to correct
behavior and I tried to document the code so it would be easier
to follow..
I frequently want to the the first N digits of an OID formatted
as a string and I'd like it to be efficient. This function makes
that easy and I could rewrite the OID formatters in terms of it.
Expose a way to retrieve, along with the target git_object, the reference
pointed at by some revparse expression (`@{<-n>}` or
`<branchname>@{upstream}` syntax).
In theory, if there was a problem reading the REUC data, the
read_reuc() routine could have left uninitialized and invalid
data in the git_index vector. This moves the line that inserts a
new entry into the vector down to the bottom of the routine so we
know all the content is already valid. Also, per @linquize, this
uses calloc to ensure no uninitialized data.
This adds an example implementation that emulates git cat-file.
It is a convenient and relatively simple example of getting data
out of a repository.
Implementing this also revealed that there are a number of APIs
that are still not using const pointers to objects that really
ought to be. The main cause of this is that `git_vector_bsearch`
may need to call `git_vector_sort` before doing the search, so a
const pointer to the vector is not allowed. However, for tree
objects, with a little care, we can ensure that the vector of
tree entries is always sorted and allow lookups to take a const
pointer. Also, the missing const in commit objects just looks
like an oversight.
Since I added the GIT_IDXENTRY_STAGE macro to extract the stage
from a git_index_entry, we probably don't need an internal inline
function to do the same thing.
Under some strange circumstances, diffs can end up listing files
that we can't actually open successfully. Instead of aborting
the git_diff_find_similar, this makes it so that those files just
won't be considered as valid rename/copy targets instead.
It is possible for there to be a submodule in a repository with
no .gitmodules file (for example, if the user forgot to commit
the .gitmodules file). In this case, core Git will just create
an empty directory as a placeholder for the submodule but
otherwise ignore it. We were generating an error and stopping
the checkout. This makes our behavior match that of core git.
Unlike blob updates, symlink updates cannot be done "in place"
writing over an old symlink. This means that in checkout when we
realize that we can safely update a symlink, we still need to
remove the old one before writing the new.
When the last item in a diff was an untracked directory that only
contained ignored items, the loop to scan the contents would run
off the end of the iterator and dereference a NULL pointer. This
includes a test that reproduces the problem and a fix.
There was a problem found in the Rugged test suite where the
refdb_fs_backend__next function could exit too early in some
very specific hashing patterns for packed refs. This ports
the Rugged test to libgit2 and then fixes the bug.
Nobody should ever be using anything other than ALL at this level, so
remove the option altogether.
As part of this, git_reference_foreach_glob is now implemented in the
frontend using an iterator. Backends will later regain the ability of
doing the glob filtering in the backend.
If you use rename detection, the renamed and copied files would
not show any text diffs because the function that decides if
data should be loaded didn't know which sides of the diff to
load for those cases.
This adds a test that looks at the patch generated for diff
entries that are COPIED or RENAMED.
The git_status_file API was doing a hack to deal with files that
are inside ignored directories. The status scan was not reporting
any file in this case, so git_status_file would attempt a final
"stat()" call, and return IGNORED if the file actually existed.
On case-insensitive filesystems where core.ignorecase is set
incorrectly, this magic check can "succeed" and report a file
as ignored when it should actually return ENOTFOUND.
Now that we have the GIT_STATUS_OPT_RECURSE_IGNORED_DIRS, we can
use that flag to make sure that git_status_file() will look into
ignored directories and eliminate the hack completely, so we give
the correct error.
This clarifies the docs for git_repository_message and also adds
to the tests to explicitly check NUL termination of data when the
output buffer is smaller than the message size. There is a minor
behavior change so that a non-NULL output buffer will always be
NUL terminated (at length zero) if an error occurs.
When a repository is initialised, we need to probe to see if there is
a global config to load. If this is not the case, the user isn't able
to write to the global config without creating the backend and adding
it themselves, which is inconvenient and overly complex.
Unconditionally create and add a backend for the global config file
regardless of whether it exists as a convenience for users.
To enable this, we allow creating backends to files that do not exist
yet, changing the semantics somewhat, and making some tests invalid.
When tagopt is set to '--tags', we should only take the default tags
refspec into account and ignore any configured ones.
Bring the code into compliance.
When a patch contained an eofnl change (i.e. the last line either
gained or lost a newline), the oldno and newno line number values
for the lines in the last hunk of the patch were not useful. This
makes them behave in a more expected manner.
This adds a new line origin constant for the special line that
is used when both files end without a newline.
In the course of writing the tests for this, I was having problems
with modifying a file but not having diff notice because it was
the same size and modified less than one second from the start of
the test, so I decided to start working on nanosecond timestamp
support. This commit doesn't contain the nanosecond support, but
it contains the reorganization of maybe_modified and the hooks so
that if the nanosecond data were being read by stat() (or rather
being copied by git_index_entry__init_from_stat), then the nsec
would be taken into account.
This new stuff could probably use some more tests, although there
is some amount of it here.
Currently git_branch_set_upstream when passed a local branch
creates invalid configuration, for ex. if we setup branch
'tracking_master' to track local 'master' libgit2 generates
the following config
```
[branch "track_master"]
remote = .
merge = .refs/heads/track_master
```
The merge value is invalid and calling git_branch_upstream on
'tracking_master' results in invalid reference error.
It should do:
```
[branch "track_master"]
remote = .
merge = refs/heads/master
```
We use p->index_map.data to check whether the struct has been set up
and all the information about the index is stored there. This variable
gets set up halfway through the setup process, however, and a thread
can come along and use fields that haven't been written to yet.
Crucially, pack_entry_find_offset() needs to read the index version
(which is written after index_map) to know the offset and stride
length to pass to sha1_entry_pos(). If these values are wrong,
assertions in it will fail, as it will be reading bogus data.
Make index_version the last field to be written and switch from using
p->index_map.data to p->index_version as "git_pack_file is ready" flag
as we can use it to know if every field has been written.
Older versions of git would only write peeled entries for
items under refs/tags/. Newer versions will write them for
all refs, and we should be prepared to handle that.
There is an occasional assertion failure in sha1_entry_pos from
pack_entry_find_index when running threaded. Holding the mutex
around the code that grabs the index_map data and processes it
makes this assertion failure go away.
There are many paths through revparse that may return an error
code without reporting an error, I believe. This fixes one of
them. Because of the backtracking in revparse, it is pretty
complicated to fix the others.
A number of places were looking up option config values and then
not clearing the error codes if the values were not found. This
moves the repeated pattern into a shared routine and adds the
extra call to giterr_clear() when needed.
There are some cases, particularly where no loaded ODB backends
support a particular operation, where we would return an error
code without having set an error. This catches those cases and
reports that no ODB backends support the operation in question.
When diff encounters an untracked directory, there was a shortcut
that it took which is not compatible with core git. This makes
the default behavior no longer take that shortcut and instead look
inside the untracked directory to see if there are any untracked
files within it. If there are not, then the directory is treated
as an ignore directory instead of an untracked directory. This
has implications for the git_status APIs.
In preparation for more changes to the internal diff logic, it
seemed wise to split the very large git_diff__from_iterators into
separate functions that handle the four main cases (unmatched old
item, unmatched new item, unmatched new directory, and matched
old and new items). Hopefully this will keep the logic easier to
follow even as more cases have to be added to this code.
This removes the GIT_INLINE versions of the simple git_object
accessors and standardizes them with a helper macro in src/object.h
to build the function bodies.
Add a new git_oid_strcmp that compares a string OID with a hex
oid for sort order, and then reimplement git_oid_streq using it.
This actually should speed up git_oid_streq because it only reads
as far into the string as it needs to, whereas previously it would
convert the whole string into an OID and then use git_oid_cmp.
git_oid_ncmp was making some assumptions about the length of
the data - this shifts the check to the top of the loop so it
will work more robustly, limits the max, and adds some tests
to verify the functionality.
For update and create commands where all the objects are known to
exist in the remote, we must send an empty packfile. However, if all
we issue are delete commands, no packfile must be sent.
Take this into consideration for push.
This makes diff use the cvar cache for config options where
possible, and also adds support for a number of other config
options to diff including "diff.context", "diff.ignoreSubmodules",
"diff.noprefix", "diff.mnemonicprefix", and "core.abbrev".
To make this natural, this involved a rearrangement of the code
that allocates the diff object vs. the code that initializes it
based on the combination of options passed in by the user and
read from the config.
This commit includes tests for most of these new options as well.
This converts many of the config lookups that are done around the
library to use the repository config cache. This was everything I
could find that wasn't part of diff (which requires a larger fix).
This adds a bunch of additional config values to the repository
config value cache and makes it easier to add a simple boolean
config without creating enum values for each possible setting.
Also, this fixes a bug in git_config_refresh where the config
cache was not being cleared which could lead to potential
incorrect values.
The work to start using the new cached configs will come in the
next couple of commits...
When case insensitive tree iterators were added, we started reading
the case sensitivity of the index to decide if the tree should be
case sensitive. This is good for index-to-tree comparisons, but
for tree-to-tree comparisons, we should really default to doing a
case sensitive comparison unless the user really wants otherwise.
Rename git_packfile_check to git_packfile_alloc since it is now
being used more in that capacity. Fix the various places that use
it. Consolidate some repeated code in odb_pack.c related to the
allocation of a new pack_backend.
The indexer was creating a packfile object separately from the
code in pack.c which was a problem since I put a call to
git_mutex_init into just pack.c. This commit updates the pack
function for creating a new pack object (i.e. git_packfile_check())
so that it can be used in both places and then makes indexer.c
use the shared initialization routine.
There are also a few minor formatting and warning message fixes.
This removes the lock from the repository object and changes the
internals to use the new atomic git__compare_and_swap to update
the _odb, _config, _index, and _refdb variables in a threadsafe
manner.
This builds on the earlier thread safety work to make it so that
setting the odb, index, refdb, or config for a repository is done
in a threadsafe manner with minimized locking time. This is done
by adding a lock to the repository object and using it to guard
the assignment of the above listed pointers. The lock is only
held to assign the pointer value.
This also contains some minor fixes to the other work with pack
files to reduce the time that locks are being held to and fix an
apparently memory leak.
This adds create and free callback to the git_objects_table so
that more of the creation and destruction of objects can be table
driven instead of using switch statements. This also makes the
semantics of certain object creation functions consistent so that
we can make better use of function pointers. This also fixes a
theoretical error case where an object allocation fails and we
end up storing NULL into the cache.
When I was writing threading tests for the new cache, the main
error I kept running into was a pack file having it's content
unmapped underneath the running thread. This adds a lock around
the routines that map and unmap the pack data so that threads can
effectively reload the data when they need it.
This also required reworking the error handling paths in a couple
places in the code which I tried to make consistent.
Add a git_cache_set_max_object_size method that does more checking
around setting the max object size. Also add a git_cache_size to
read the number of objects currently in the cache. This makes it
easier to write tests.
This moves most of the refdb stuff over to the include/git2/sys
directory, with some minor shifts in function organization.
While I was making the necessary updates, I also removed the
trailing whitespace in a few files that I modified just because I
was there and it was bugging me.
Actually this renames git_commit_create_oid to
git_commit_create_from_oids and moves the API declaration to
include/git2/sys/commit.h since it is a dangerous API for general
use (because it doesn't check that the OID list items actually
refer to real objects).
This moves some of the odb_backend stuff that is related to the
internals of an odb_backend implementation into include/git2/sys.
Some of the stuff related to streaming I left in include/git2
because it seemed like it would be reasonably needed by a normal
user who wanted to stream objects into and out of the ODB.
Also, I added APIs for traversing the list of backends so that
some of the tests would not need to access ODB internals.
It used to be separate as an attempt to make the querying easier, but
it didn't work out that way, so put all the data together.
Add git_refspec_string() as well to get the original string, which is
now stored alongside the independent parts.
Introduce git_remote_{fetch,push}_refspecs() to get a list of refspecs
from the remote and rename the refspec-adding functions to a less
silly name.
Use this instead of the vector index hacks in the tests.
A remote can have a multitude of refspecs. Up to now our git_remote's
have supported a single one for each fetch and push out of simplicity
to get something working.
Let the remotes and internal code know about multiple remotes and get
the tests passing with them.
Instead of setting a refspec, the external users can clear all and add
refspecs. This should be enough for most uses, though we're still
missing a querying function.
This adds some hooks into the filesystem iterator so that the
workdir iterator can just become a wrapper around it. Then we
remove most of the workdir iterator code and just have it augment
the filesystem iterator with skipping .git entries, updating the
ignore stack, and checking for submodules.
This adds a new variant iterator that is a raw filesystem iterator
for scanning directories from a root. There is still more work to
do to blend this with the working directory iterator.
If the on-disk file has been staged (it's stat data matches the stat data
in the cache) then we need not hash the file to determine whether it
differs from the checkout target; instead we can simply use the oid in
the index.
This prevents recomputing a file's hash unnecessarily, prevents loading
the file (when filtering) and prevents edge cases where filters suggest
that a file is dirty immediately after git writes the file.
Don't try to update anything if there are no heads to update. This
saves us from trying to look into a fetch refspec when there is none.
A better fix for compatibility with git when using remotes without
refspecs is still needed, but this stops us from segfaulting.
The end of the header is signaled by to consecutive LFs and the commit
message starts immediately after. Jumping over LFs at the start of the
message is a bug and leads to creating different commits if
when rebuilding history.
This also fixes an empty commit message being returned as "\n".
This adds tests for diffs with submodules in them and (perhaps
unsurprisingly) requires further fixes to be made. Specifically,
this fixes:
- when considering if a submodule is dirty in the workdir, it was
being treated as dirty even if only the index was dirty.
- git_diff_patch_to_str (and git_diff_patch_print) were "printing"
the headers for files (and submodules) that were unmodified or
had no meaningful content.
- added comment to previous fix and removed unneeded parens.
GIT_DIFF_PATCH_DIFFABLE was not set, so the diff content was not shown
When submodule is dirty, the hash may be the same, but the length is different because -dirty is appended
We can therefore compare the length or hash
Return the size we'd need to write to instead of simply an
error. Split the function into two to be used later by the upstream
configuration functions.
This started out trying to look at the problems from issue #1425
and gradually grew to a broader set of fixes. There are two core
things fixed here:
1. When you had an ignore like "/bin" which is rooted at the top
of your tree, instead of immediately adding the "bin/" entry
as an ignored item in the diff, we were returning all of the
direct descendants of the directory as ignored items. This
changes things to immediately ignore the directory. Note that
this effects the behavior in test_status_ignore__subdirectories
so that we no longer exactly match core gits ignore behavior,
but the new behavior probably makes more sense (i.e. we now
will include an ignored directory inside an untracked directory
that we previously would have left off).
2. When a submodule only contained working directory changes, the
diff code was always considering it unmodified which was just
an outright bug. The HEAD SHA of the submodule matches the SHA
in the parent repo index, and since the SHAs matches, the diff
code was overwriting the actual status with UNMODIFIED.
These fixes broke existing tests test_diff_workdir__submodules and
test_status_ignore__subdirectories but looking it over, I actually
think the new results are correct and the old results were wrong.
@nulltoken had actually commented on the subdirectory ignore issue
previously.
I also included in the tests some debugging versions of the
shared iteration callback routines that print status or diff
information. These aren't used actively in the tests, but can be
quickly swapped in to test code to give a better picture of what
is being scanned in some of the complex test scenarios.
This option has been sitting unimplemented for a while, so I
finally went through and implemented it along with some tests.
As part of this, I improved the implementation of
GIT_DIFF_IGNORE_SUBMODULES so it be more diligent about avoiding
extra work and about leaving off delta records for submodules to
the greatest extent possible (though it may include them still
if you are request TYPECHANGE records).
This implements working versions of GIT_DIFF_RECURSE_IGNORED_DIRS
and GIT_STATUS_OPT_RECURSE_IGNORED_DIRS along with some tests for
the newly available behaviors. This is not turned on by default
for status, but can be accessed via the options to the extended
version of the command.
This adds crlf/lf conversion functions into buf_text with more
efficient implementations that bypass the high level buffer
functions. They attempt to minimize the number of reallocations
done and they directly write the buffer data as needed if they
know that there is enough memory allocated to memcpy data.
Tests are added for these new functions. The crlf.c code is
updated to use the new functions.
Removed the include of buf_text.h from filter.h and just include
it more narrowly in the places that need it.
This adds a check to the drop_crlf filter path to check it the
file in the index already has a CR in it, in which case this will
not drop the CRs from the workdir file contents.
This uncovered a "bug" in `git_blob_create_fromworkdir` where the
full path to the file was passed to look up the attributes instead
of the relative path from the working directory root. This meant
that the check in the index for a pre-existing entry of the same
name was failing.
Currently, the odb cache has a fixed size of 128 slots as defined by
GIT_DEFAULT_CACHE_SIZE. Allow users to set the size of the cache via
git_libgit2_opts().
Fixes#1035.
1. Fix sort order problem with submodules where "mod" was sorting
after "mod-plus" because they were being sorted as "mod/" and
"mod-plus/". This involved pushing the "contains a .git entry"
test significantly lower in the stack.
2. Reinstate behavior that a directory which contains a .git entry
will be treated as a submodule during iteration even if it is
not yet added to the .gitmodules.
3. Now that any directory containing .git is reported as submodule,
we have to be more careful checking for GIT_EEXISTS when we
do a submodule lookup, because that is the error code that is
returned by git_submodule_lookup when you try to look up a
directory containing .git that has no record in gitmodules or
the index.
This switches the APIs for setting and getting the global/system
search paths from using git_strarray to using a simple string with
GIT_PATH_LIST_SEPARATOR delimited paths, just as the environment
PATH variable would contain. This makes it simpler to get and set
the value.
I also added code to expand "$PATH" when setting a new value to
embed the old value of the path. This means that I no longer
require separate actions to PREPEND to the value.
Implicit type conversion argument of function to size_t type
Suspicious sequence of types castings: size_t -> int -> size_t
Consider reviewing the expression of the 'A = B == C' kind. The expression is calculated as following: 'A = (B == C)'
Unsigned type is never < 0
The goal of this work is to expose the search logic for "global",
"system", and "xdg" files through the git_libgit2_opts() interface.
Behind the scenes, I changed the logic for finding files to have a
notion of a git_strarray that represents a search path and to store
a separate search path for each of the three tiers of config file.
For each tier, I implemented a function to initialize it to default
values (generally based on environment variables), and then general
interfaces to get it, set it, reset it, and prepend new directories
to it.
Next, I exposed these interfaces through the git_libgit2_opts
interface, reusing the GIT_CONFIG_LEVEL_SYSTEM, etc., constants
for the user to control which search path they were modifying.
There are alternative designs for the opts interface / argument
ordering, so I'm putting this phase out for discussion.
Additionally, I ended up doing a little bit of clean up regarding
attr.h and attr_file.h, adding a new attrcache.h so the other two
files wouldn't have to be included in so many places.
This adds a git_pool_freelist_item struct that makes it a little
easier to follow what's going on with the pool free list block
management code. It is functionally neutral.
This fixes a number of issues identified by valgrind - mostly
missed free calls. Inside valgrind, mmap() may fail which causes
some of the diff tests to fail. This adds a fallback code path
to diff_output.c:get_workdir_content() where is the mmap() fails
the code will now try to read the file data directly into allocated
memory (which is what it would do if the data needed to be filtered
anyhow).
This updates the tree iterator internals to be more efficient.
The tree_iterator_entry objects are now kept as pointers that are
allocated from a git_pool, so that we may use git__tsort_r for
sorting (which is better than qsort, given that the tree is
likely mostly ordered already).
Those tree_iterator_entry objects now keep direct pointers to the
data they refer to instead of keeping indirect index values. This
simplifies a lot of the data structure traversal code.
This also adds bsearch to find the start item position for range-
limited tree iterators, and is more explicit about using
git_path_cmp instead of reimplementing it. The git_path_cmp
changed a bit to make it easier for tree_iterators to use it (but
it was barely being used previously, so not a big deal).
This adds a git_pool_free_array function that efficiently frees a
list of pool allocated pointers (which the tree_iterator keeps).
Also, added new tests for the git_pool free list functionality
that was not previously being tested (or used).
This fixes two bugs with the workdir iterator depth check: first
that the depth was not being decremented and second that empty
directories were counting against the depth even though a frame
was not being created for them.
This also fixes a bug with the ENOTFOUND return code for workdir
iterators when you attempt to advance_into an empty directory.
Actually, that works correctly, but it was incorrectly being
propogated into regular advance() calls in some circumstances.
Added new tests for the above that create a huge hierarchy on
the fly and try using the workdir iterator to traverse it.
Clean up some sorting function stuff including fixing qsort_r
on MinGW, common function pointer type for comparison, and basic
insertion sort implementation (which we, regrettably, fall back
on for MinGW).
Given a group of case-insensitively equivalent tree iterator
entries, this ensures that the case-sensitively first trees will
be used as the representative items. I.e. if you have conflicting
entries "A/B/x", "a/b/x", and "A/b/x", this change ensures that
the earliest entry "A/B/x" will be returned. The actual choice
is not that important, but it is nice to have it stable and to
have it been either the first or last item, as opposed to a
random item from within the equivalent span.
Tree iterator advance was moving forward without taking the
filemode of the entries into account, equating "a" and "a/".
This makes the tree entry comparison code more easily reusable
and fixes the problem.
This fixes an off by one error for generating full paths for
tree entries in tree iterators when INCLUDE_TREES is set. Also,
contains a bunch of small code cleanups with a couple of small
utility functions and macro changes to eliminate redundant code.
If there are case-ambiguities in the path of a case insensitive
tree iterator, it will now rewrite the entire path when it gives
the path name to an entry, so a tree with "A/b/C/d.txt" and
"a/B/c/E.txt" will give the true full paths (instead of case-
folding them both to "A/B/C/d.txt" or "a/b/c/E.txt" or something
like that.
Previously, 0 meant default. This is problematic, as asking for 0
context lines is a valid thing to do.
Change GIT_DIFF_OPTIONS_INIT to default to three and stop treating 0
as a magic value. In case no options are provided, make sure the
options in the diff object default to 3.
Passing NULL is non-sensical. The error message leaves to be desired,
though, as it leaks internal implementation details. Catch it at the
`git_config_set_string` level and set an appropriate error message.
There is a serious bug in the previous tree iterator implementation.
If case insensitivity resulted in member elements being equivalent
to one another, and those member elements were trees, then the
children of the colliding elements would be processed in sequence
instead of in a single flattened list. This meant that the tree
iterator was not truly acting like a case-insensitive list.
This completely reworks the tree iterator to manage lists with
case insensitive equivalence classes and advance through the items
in a unified manner in a single sorted frame.
It is possible that at a future date we might want to update this
to separate the case insensitive and case sensitive tree iterators
so that the case sensitive one could be a minimal amount of code
and the insensitive one would always know what it needed to do
without checking flags.
But there would be so much shared code between the two, that I'm
not sure it that's a win. For now, this gets what we need.
More tests are needed, though.
It's somewhat common to try to write "/refs/tags/something". There is
no easy way to catch it during the main body of the function, as there
is no way to distinguish whether it's a leading slash or a double
slash somewhere in the middle.
Catch this at the beginning so we don't trigger the assert in
is_all_caps_and_underscore().
This standardizes iterator behavior across all three iterators
(index, tree, and working directory). Previously the working
directory iterator behaved differently from the other two.
Each iterator can now operate in one of three modes:
1. *No tree results, auto expand trees* means that only non-
tree items will be returned and when a tree/directory is
encountered, we will automatically descend into it.
2. *Tree results, auto expand trees* means that results will
be given for every item found, including trees, but you
only need to call normal git_iterator_advance to yield
every item (i.e. trees returned with pre-order iteration).
3. *Tree results, no auto expand* means that calling the
normal git_iterator_advance when looking at a tree will
not descend into the tree, but will skip over it to the
next entry in the parent.
Previously, behavior 1 was the only option for index and tree
iterators, and behavior 3 was the only option for workdir.
The main public API implications of this are that the
`git_iterator_advance_into()` call is now valid for all
iterators, not just working directory iterators, and all the
existing uses of working directory iterators explicitly use
the GIT_ITERATOR_DONT_AUTOEXPAND (for now).
Interestingly, the majority of the implementation was in the
index iterator, since there are no tree entries there and now
have to fake them. The tree and working directory iterators
only required small modifications.
The iterator APIs are not currently consistent with the parameter
ordering of the rest of the codebase. This rearranges the order
of parameters, simplifies the naming of a number of functions, and
makes somewhat better use of macros internally to clean up the
iterator code.
This also expands the test coverage of iterator functionality,
making sure that case sensitive range-limited iteration works
correctly.
`git_diff_get_patch()` would unconditionally load the patch object and
then simply leak it if the user hadn't requested it. Short-circuit
loading the object if the user doesn't want it.
The rest of the plugs are simply calling the free functions of objects
allocated during the tests.
These offsets are needed for REF_DELTA objects, which encode which
object they use as a base, but not where it lies in the packfile, so
we need a list.
These objects are mostly from older packfiles, before OFS_DELTA was
widely spread. The time spent in indexing these packfiles is greatly
reduced, though remains above what git is able to do.
This was the first implementation and its goal was simply to have
something that worked. It is slow and now it's just taking up
space. Remove it and switch the one known usage to use the streaming
indexer.
This removes assertions that prevent us from having an empty
git_config object and then updates some tests that were
dependent on global config state to use an empty config before
running anything.
This removes the one-off GIT_CDECL and adds a new standard way of
doing this named GIT_STDLIB_CALL with a src/win32 specific def
when on the Windows platform.
When creating files, instead of actually using GIT_FILEMODE_BLOB
and the other various constants that happen to correspond to
mode values, apparently I should be just using 0666 and 0777, and
relying on the umask to clear bits and make the value sane.
This fixes the rules for copying a template directory and fixes
the checks to match that new behavior. (Further changes to the
checkout logic to follow separately.)
The new tests were not taking core.filemode into account when
testing file modes after repo initialization. Fixed that and some
other Windows warnings that have crept in.
When PR #1359 removed the hooks from the test resources/template
directory, it made me realize that the tests for
git_repository_init_ext using templates must be pretty shabby
because we could not have been testing if the hooks were getting
created correctly.
So, this started with me recreating a couple of hooks, including
a sample and symlink, and adding tests that they got created
correctly in the various circumstances, including with the SHARED
modes, etc. Unfortunately this uncovered some issues with how
directories and symlinks were copied and chmod'ed. Also, there
was a FIXME in the code related to the chmod behavior as well.
Going back over the directory creation logic for setting up a
repository, I found it was a little difficult to read and could
result in creating and/or chmod'ing directories that the user
almost certainly didn't intend.
So that let to this work which makes repo initialization much
more careful (and hopefully easier to follow). It required a
couple of extensions / changes to core fileops utilities, but I
also think those are for the better, at least for git_futils_cp_r
in terms of being careful about what actions it takes.
This is designed to fix libgit2sharp #350 where if .gitignore is
a directory we abort all operations that process ignores instead
of just skipping it as core git does.
Also added test that fails without this change and passes with it.
This moves a couple of checks outside of the inner loop of the
find_similar rename/copy detection phase that are only dependent
on the "from" side of a detection.
Also, this replaces the inefficient initialization of the
options structure when a value is not provided explicitly by the
user.
Instead of creating three git_diff_similarity_metric statically
for the various config options, just create the metric structure
on demand and populate it, using the payload to specific the
extra flags that should be passed to the hashsig. This removes
a level of obfuscation from the code, I think.
This adds some new tests that actually exercise the similarity
metric between files to detect renames, copies, and split modified
files that are too heavily modified.
There is still more testing to do - these tests are just partially
covering the cases.
There is also one bug fix in this where a change set with only
MODIFY being broken into ADD/DELETE (due to low self-similarity)
without any additional RENAMED entries would end up not processing
the split requests (because the num_rewrites counter got reset).
This is the initial integration of the similarity metric into
the `git_diff_find_similar()` code path. The existing tests all
pass, but the new functionality isn't currently well tested. The
integration does go through the pluggable metric interface, so it
should be possible to drop in an alternative to the internal
metric that libgit2 implements.
This comes along with a behavior change for an existing interface;
namely, passing two NULLs to git_diff_blobs (or passing NULLs to
git_diff_blob_to_buffer) will now call the file_cb parameter zero
times instead of one time. I know it's strange that that change
is paired with this other change, but it emerged from some
initialization changes that I ended up making.
Previously the git_diff_delta recorded if the delta was binary.
This replaces that (with no net change in structure size) with
a full set of flags. The flag values that were already in use
for individual git_diff_file objects are reused for the delta
flags, too (along with renaming those flags to make it clear that
they are used more generally).
This (a) makes things somewhat more consistent (because I was
using a -1 value in the "boolean" binary field to indicate unset,
whereas now I can just use the flags that are easier to understand),
and (b) will make it easier for me to add some additional flags to
the delta object in the future, such as marking the results of a
copy/rename detection or other deltas that might want a special
indicator.
While making this change, I officially moved some of the flags that
were internal only into the private diff header.
This also allowed me to remove a gross hack in rename/copy detect
code where I was overwriting the status field with an internal
value.
This plugs in the three basic similarity strategies for handling
whitespace via internal use of the pluggable API. In so doing, I
realized that the use of git_buf in the hashsig API was not needed
and actually just made it harder to use, so I tweaked that API as
well.
Note that the similarity metric is still not hooked up in the
find_similarity code - this is just setting out the function that
will be used.
This moves the similarity metric code out of buf_text and into a
new file. Also, this implements a different approach to similarity
measurement based on a Rabin-Karp rolling hash where we only keep
the top 100 and bottom 100 hashes. In theory, that should be
sufficient samples to given a fairly accurate measurement while
limiting the amount of data we keep for file signatures no matter
how large the file is.
This makes the text similarity metric treat \r as equivalent
to \n and makes it skip whitespace immediately following a line
terminator, so line indentation will have less effect on the
difference measurement (and so \r\n will be treated as just a
single line terminator).
This also separates the text and binary hash calculators into
two separate functions instead of have more if statements inside
the loop. This should make it easier to have more differentiated
heuristics in the future if we so wish.
This adds a new `git_buf_text_hashsig` type and functions to
generate these hash signatures and compare them to give a
similarity score. This can be plugged into diff similarity
scoring.
This replaces most of the explicit vector iteration with calls
to git_vector_foreach, adds in some git__free and giterr_clear
calls to clean up during some error paths, and a couple of
other code simplifications.
The treebuilder entries vector flags removed items which means
we can't rely on the entries vector length to accurately get the
number of entries. This adds an entrycount value and maintains it
while updating the treebuilder entries.
The cppcheck static analyzer generates warnings for a bunch of
places in the libgit2 code base. All the ones fixed in this
commit are actually false positives, but I've reorganized the
code to hopefully make it easier for static analysis tools to
correctly understand the structure. I wouldn't do this if I
felt like it was making the code harder to read or worse for
humans, but in this case, these fixes don't seem too bad and will
hopefully make it easier for better analysis tools to get at any
real issues.
If gethostbyname() fails on platforms with NO_ADDRINFO, the code
leaks the struct addrinfo that was allocated. This fixes that
(and a number of code formatting issues in that area of code in
src/posix.c).
`git_diff_blobs` and `git_diff_blob_to_buffer` skip the step
where we check file attributes because they don't have a filename
associated with the data. Unfortunately, this meant they were also
skipping the check for the GIT_DIFF_FORCE_TEXT option and so you
could not force a diff of an apparent binary file. This adds the
force text check into their code path.
The callback will be called for each file, just before the `git_delta_t` gets inserted into the diff list.
When the callback:
- returns < 0, the diff process will be aborted
- returns > 0, the delta will not be inserted into the diff list, but the diff process continues
- returns 0, the delta is inserted into the diff list, and the diff process continues
Instead of returning directly the pattern as the return value, I used an
out parameter, because the function also tests if the passed pathspecs
vector is empty. If yes, it considers that the path "matches", but in
that case there is no matched pattern per se.
W/o this a libgit2 error message could have a mixed encoding:
e.g. a filename in UTF-8 combined with a native Windows error message
encoded with the local code page.
Signed-off-by: Sven Strickroth <email@cs-ware.de>
A leading slash confuses the name normalization code when the flags
include ALLOW_ONELEVEL. Catch this case in particular to avoid
triggering an assertion in the uppercase check which expects us not to
pass it an empty string.
The existing tests don't catch this as they simply use the NORMAL
flag.
This fixes#1300.
This adds a `git_diff_patch_line_stats()` API that gets the total
number of adds, deletes, and context lines in a patch. This will
make it a little easier to emulate `git diff --stat` and the like.
Right now, this relies on generating the `git_diff_patch` object,
which is a pretty heavyweight way to get stat information. At
some future point, it would probably be nice to be able to get
this information without allocating the entire `git_diff_patch`,
but that's a much larger project.
This is a new implementation of core git's config key checking
rules that prevents non-alphanumeric characters (and '-') for
the top-level section and key names inside of config files.
This also validates the target section name when renaming
sections.
OpenBSD's realpath(3) doesn't require the last part of the path to
exist. Override p_realpath in this OS to bring it in line with the
library's assumptions.
Check whether the backslash at the end of the line is being escaped or
not so as not to consider it a continuation marker when it's e.g. a
Windows-style path.
This is a convenience function to get the branch name of a given
ref. The returned branch name is compatible with the name that can
be supplied e.g. to git_branch_lookup(). That is, the prefixes
"refs/heads" or "refs/remotes" are omitted.
Also added a new test for testing the new function.
With the new code to make tree iterators support ignore_case,
there is a bug in setting the start entry for range bounded
iterators where memcmp was being used instead of strncasecmp.
This fixes that and expands the tree iterator test to cover
the cases that were broken.
The commit time is already stored as a git_time_t, but we were
parsing is as a uint32_t. This just switches the parser to use
uint64_t which will handle dates further in the future (and adds
some tests of those future dates).
When the encoding header changed to be treated as an additional
header, the EOL pointer started to point to the byte after the LF,
making the git__strndup call copy the LF into the value.
Increase the EOL pointer value after copying the data to keep the rest
of the semantics but avoid copying LF.
This moves the check for the "encoding" header into a loop which
is just scanning for non-required headers at the end of a commit
header. That loop will skip unrecognized lines (including header
continuation lines) until a terminating completely blank line is
found, and only then does it move to reading the commit message.
This makes tree iterators directly support case insensitivity by
using a secondary index that can be sorted by icase. Also, this
fixes the ambiguity check in the git_status_file API to also be
case insensitive. Lastly, this adds new test cases for case
insensitive range boundary checking for all types of iterators.
With this change, it should be possible to deprecate the spool
and sort iterator, but I haven't done that yet.
This adds a new external API git_tree_entry_cmp and a new internal
API git_tree_entry_icmp for sorting tree entries. The case
insensitive one is internal only because general users should
never be seeing case-insensitively sorted trees.
git__bsearch and git__tsort did not pass a payload through to the
comparison function. This makes it impossible to implement sorted
lists where the sort order depends on external data (e.g. building
a secondary sort order for the entries in a tree). This commit
adds git__bsearch_r and git__tsort_r versions that pass a third
parameter to the cmp function of a user payload.
This changes the iterator API so that flags can be passed in to
the constructor functions to control the ignore_case behavior.
At this point, the flags are not supported on tree iterators (i.e.
there is no functional change over the old API), but the API
changes are all made to accomodate this.
By the way, I went with a flags parameter because in the future
I have a couple of other ideas for iterator flags that will make
it easier to fix some diff/status/checkout bugs.
Returning GIT_EAMBIGUOUS from inside the status callback gets
overridden with GIT_EUSER. `git_status_file` accounted for this
via the callback payload, but was allowing the error message to
be cleared. Move the `giterr_set` call outside the callback to
where the EUSER case was being dealt with.
In preparation for further iterator changes, this cleans up a few
small things in the iterator API:
* removed the git_iterator_for_repo_index_range API
* made git_iterator_free not be inlined
* minor param name and test function name tweaks
Somewhat surprisingly, this can increase the speed considerably, as we
don't bother trying to decide what to evict, and the most used entries
are quickly back into the cache.
This is an intermin solution. While this essentially disables the
--shared flag feature, previously external templates did not work
at all. This change fixes the previously corrected, and since
then failing, repo_init__extended_with_template() test.
The problem is now documented in the source code comments.
The indexer needs to call the packfile's free function so it takes care of
freeing the caches.
We still need to close the mwf descriptor manually so we can rename the
packfile into its final name on Windows.
Core git just looks for NUL bytes in files when deciding about
is-binary inside diff (although it uses a better algorithm in
checkout, when deciding if CRLF conversion should be done).
Libgit2 was using the better algorithm in both places, but that
is causing some confusion. For now, this makes diff just look
for NUL bytes to decide if a file is binary by content in diff.
This was just wrong. Added a test that verifying patch line
numbers even for hunks further into a file and then fixed the
algorithm. I needed to add a little extra state into the patch
so that I could track old and new file numbers independently,
but it should be okay.
Many delta bases are re-used. Cache them to avoid inflating the same
data repeatedly.
This version doesn't limit the amount of entries to store, so it can
end up using a considerable amound of memory.
This adds an option to checkout a la the diff option to turn off
fnmatch evaluation for pathspec entries. This can be useful to
make sure your "pattern" in really interpretted as an exact file
match only.
All the ODB backends have a specific refresh interface. When reading an
object, first we attempt every single backend: if the read fails, then
we refresh all the backends and retry the read one more time to see if
the object has appeared.
It is not legal inside our `p_mmap` function to mmap a zero length
file. This adds a test that exercises that case inside diff and
fixes the code path where we would try to do that.
The fix turns out not to be a lot of code since our default file
content is already initialized to "" which works in this case.
Fixes#1210
This moves the implementation of these two APIs into common code
that will be shared between the two. Also, this adds tests for
the `git_diff_blob_to_buffer` API. Lastly, this adds some extra
`const` to a few places that can use it.
Before this, we error out from `reference_matches_remote_head` if the
reference we're searching for does not exist.
Since we explicitly check if master is existing in `update_head_to_remote`
and error out if it doesn't, a repository without master branch could
not be cloned.
In fact this was later clobbered by what is fixed in #1194.
However, this patch introduces a `found` member in `head_info` and sets
it accordingly. That also saves us from checking the string length of
`branchname` a few times.
As a function that appears to only be called on error paths, I don't
think it makes sense for it to return an error, or clobber the global
giterr. Note that no existing callsites actually check the return
code.
In my own application, there are errors where the real error ends
up being hidden, as git_mwindow_file_deregister() clobbers the
global giterr. I'm not sure this error is even relevant?
I saw a repo in the wild today which had a master branch ref which was packed, but had no trailing newline. Git handled it fine, but libgit2 choked on it. Fix seems simple enough. If we don't see a newline, assume the end of the buffer is the end of the ref line.
There are a couple of checkout bugs fixed here. One is with
untracked working directory entries that are prefixes of tree
entries but not in a meaningful way (i.e. "read" is a prefix of
"readme.txt" but doesn't interfere in any way). The second bug
is actually a redo of 07edfa0fc640f85f95507c3101e77accd7d2bf0d
where directory entries in the index that are not in the diff
were not being removed correctly. That fix remedied one case
but broke another.
When checking out with the GIT_CHECKOUT_REMOVE_UNTRACKED option
and there was an entire tree in the working directory and in the
index that is not in the baseline nor target commit, the tree was
correctly(?) removed from the working directory but was not
successfully removed from the index. This fixes that and adds a
test of the functionality.
This moves a lot of the detailed checkout documentation into a new
file (docs/checkout-internals.md) and simplifies the public docs
for the checkout API.
There were a bunch of small bugs in the checkout code where I was
assuming that a typechange was always from a tree to a blob or
vice versa. This fixes up most of those cases. Also, there were
circumstances where the submodule definitions were changed by the
checkout and the submodule data was not getting reloaded properly
before the new submodules were checked out.
The notifications were broken from the various iterations over
this code and were not returning working dir item data correctly.
Also, workdir items that were alphabetically after the last item
in diff were not being processed.
The spoolandsort iterator changes got sort-of cherry picked out of
this branch and so I dropped the commit when rebasing; however,
there were a few small changes that got dropped as well (since the
version merged upstream wasn't quite the same as what I dropped).
This adds a new API to the submodule interface that just returns
where information about the submodule was found (e.g. config file
only or in the HEAD, index, or working directory).
Also, the old "refresh" call was potentially keeping some stale
submodule data around, so this simplfies that code and literally
discards the old cache, then reallocates.
Stash was sometimes obscuring the actual error code, replacing it
with a -1 when there was more descriptive value. This updates
stash to preserve the original error code more reliably along
with a variety of other error handling tweaks.
I believe this is an improvement, but arguably, preserving the
underlying error code may result in values that are harder to
interpret by the caller who does not understand the internals.
Discussion is welcome!
Previously a NULL oid was handled like an empty buffer and
returned a status empty string. This makes git_oid_tostr()
set the output buffer to the empty string instead.
Make checkout update entries in the index for all files that are
updated and/or removed, unless flag GIT_CHECKOUT_DONT_UPDATE_INDEX
is given. To do this, iterators were extended to allow a little
more introspection into the index being iterated over, etc.
This flips checkout back to be driven off the changes between
the baseline and the target trees. This reinstates the complex
code for tracking the contents of the working directory, but
overall, I think the resulting logic is easier to follow.
I've tried to map out the detailed behaviors of checkout and make
sure that we're handling the various cases correctly, along with
providing options to allow us to emulate "git checkout" and "git
checkout-index" with the various flags. I've thrown away flags
in the checkout API that seemed like clutter and added some new
ones. Also, I've converted the conflict callback to a general
notification callback so we can emulate "git checkout" output and
display "dirty" files.
As of this commit, the new behavior is not working 100% but some
of that is probably baked into tests that are not testing the
right thing. This is a decent snapshot point, I think, along the
way to getting the update done.
This corrects the order of operations in git reset so that the
checkout to reset the working directory content is done before
the HEAD is moved. This allows us to use the HEAD and the index
content to know what files can / should safely be reset.
Unfortunately, there are still some cases where the behavior of
this revision differs from core git. Notable, a file which has
been added to the index but is not present in the HEAD is
considered to be tracked by core git (and thus removable by a
reset command) whereas since this loads the target state into
the index prior to resetting, it will consider such a file to be
untracked and won't touch it. That is a larger fix that I'll
defer to a future commit.
* gen_pktline() in smart_protocol.c was skipping refspecs that deleted
refs that were not advertised by the server. The new behavior is to
send a delete command with an old-id of zero, which matches the behavior
of the official git client.
* Update test_network_push__delete() in reaction to above fix.
* Obviate messy logic that handles missing push_spec rrefs by canonicalizing
push_spec. After calculate_work(), loid, roid, and rref, are filled in with
exactly what is sent to the server
The original libpqueue file were licensed under Apache 2.0 so
therefore should retain their copyrights and header as per the
license terms at http://www.apache.org/licenses/LICENSE-2.0
When normalizing a reference name, if there is an error because
the name is invalid, then the memory allocated for storing the
name could be leaked if the caller was not careful and assumed
that the error return code meant that no allocation had occurred.
This fixes that by explicitly deallocating the reference name
buffer if there is an error in normalizing the name.
An earlier change to `git_diff_from_iterators` introduced a
memory leak where the allocated spoolandsort iterator was not
returned to the caller and thus not freed.
One proposal changes all iterator APIs to use git_iterator** so
we can reallocate the iterator at will, but that seems unexpected.
This commit makes it so that an iterator can be changed in place.
The callbacks are isolated in a separate structure and a pointer
to that structure can be reassigned by the spoolandsort extension.
This means that spoolandsort doesn't create a new iterator; it
just allocates a new block of callbacks (along with space for its
own extra data) and swaps that into the iterator.
Additionally, since spoolandsort is only needed to switch the
case sensitivity of an iterator, this simplifies the API to only
take the ignore_case boolean and to be a no-op if the iterator
already matches the requested case sensitivity.
The diff constructor functions had some confusing names, where the
"old" side of the diff was coming after the "new" side. This
reverses the order in the function name to make it less confusing.
Specifically...
* git_diff_index_to_tree becomes git_diff_tree_to_index
* git_diff_workdir_to_index becomes git_diff_index_to_workdir
* git_diff_workdir_to_tree becomes git_diff_tree_to_workdir
According to man 3 SSL_shutdown / TLS, "If a unidirectional shutdown is
enough (the underlying connection shall be closed anyway), this first
call to SSL_shutdown() is sufficient."
Currently, an unidirectional shutdown is enough, since
gitno_ssl_teardown is called by gitno_close only. Do so to avoid further
errors (by misbehaving peers for example).
Fixes#1129.
While C Git has been writing entry count -1 (ie. never other negative
numbers) as invalid since day 1, it accepts all negative entry counts
as invalid. JGit follows the same rule. libgit2 should also follow, or
the index that works with C Git or JGit may someday be rejected by
libgit2.
Other reimplementations like dulwich and grit have not bothered with
parsing or writing tree cache.
The `git_iterator_reset` command has not been working in all cases
particularly when there is a start and end range. This fixes it
and adds tests for it, and also extends it with the ability to
update the start/end range strings when an iterator is reset.
This removes the need to explicitly pass the repo into iterators
where the repo is implied by the other parameters. This moves
the repo to be owned by the parent struct. Also, this has some
iterator related updates to the internal diff API to lay the
groundwork for checkout improvements.
If commit timestamps are off, we're more likely to hit a traversal
where the first path ends up traversing past the root commit of the tree.
If that happens, it's possible that the loop will complete before the second
path marks some of those final parents. This fix keeps track of the root
nodes that are encountered in the traversal, and verify that they are
properly marked.
In the best case, with accurate timestamps, the traversal will continue
to terminate when all the commits are STALE (parents of a merge-base), as
it did before. In the worst case, where one path makes a complete traversal
past a root commit, we will continue the loop until the root commit itself
is marked.
This could also use PTHREAD_MUTEX_INITIALIZER, but a dynamic initializer seems like a more portable concept, and we won't need another #define on top of git_mutex_init()
Storing 4kB or 8kB in the stack is not very gentle. As this part has
to be linear, put the buffer into the indexer object so we allocate it
once in the heap.
There are many different broken filemodes in the wild so we need to
protect against them and give something useful up the chain. Don't
fail when reading a tree from the ODB but normalize the mode as best
we can.
As 664 is no longer a mode that we consider to be valid and gets
normalized to 644, we can stop accepting it in the treebuilder. The
library won't expose it to the user, so any invalid modes are a bug.
To paraphrase @peff:
You can get both size and type from a packed object reasonably cheaply.
If you have:
* An object that is not a delta; both type and size are available in the
packfile header.
* An object that is a delta. The packfile type will be OBJ_*_DELTA, and
you have to resolve back to the base to find the real type. That means
potentially a lot of packfile index lookups, but each one is
relatively cheap. For the size, you inflate the first few bytes of the
delta, whose header will tell you the resulting size of applying the
delta to the base.
For simplicity, we just decompress the whole delta for now.
A mmap-window is not guaranteed to give you the whole object, but the
indexer currently assumes so.
Loop asking for more data until we've successfully CRC'd all of the
packed data.
Up to now, deltas needed to be enterily in the packfile, and we tried
to decompress then in their entirety over and over again.
Adjust the logic so we read them as they come, just as we do for full
objects. This also allows us to simplify the logic and have less
nested code. The delta resolving phase still needs to decompress the
whole object into memory, as there is not yet any streaming
delta-apply support, but it helps in speeding up the downloading
process and reduces the amount of memory allocations we need to do.
The new API allows us to read the object bit by bit from the packfile,
instead of needing it all at once in the packfile. This also allows us
to hash the object as it comes in from the network instead of having
to try to read it all and failing repeatedly for larger objects.
This is only the first step, but it already shows huge improvements
when dealing with objects over a few megabytes in size. It reduces the
memory needs in some cases, but delta objects still need to be
completely in memory and the old inefficent method is still used for
that.
`revwalk.h:commit_lookup()` -> `git_revwalk__commit_lookup()`
and make `git_commit_list_parse()` do real error checking that
the item in the list is an actual commit object. Also fixed an
apparent typo in a test name.
Moved it into graph.{c,h} which i created for the new "graph"
functions namespace. Also adjusted the function prototype
to use `size_t` and `const git_oid *`.
There are many scattered functions that look into the contents of
buffers to do various text manipulations (such as escaping or
unescaping data, calculating text stats, guessing if content is
binary, etc). This groups all those functions together into a
new file and converts the code to use that.
This has two enhancements to existing functionality. The old
text stats function is significantly rewritten and the BOM
detection code was extended (although largely we can't deal with
anything other than a UTF8 BOM).
clang-SVN HEAD kindly provided my the info, that sm_repo maybe
uninitialized when we want to free it (If the expression in line 358 or
359/360 evaluate to true, we jump to "cleanup", where we'd use sm_repo
uninitialized).
This fixes some missed places where we can apply const-ness to
various public APIs.
There are still some index and tree APIs that cannot take const
pointers because we sort our `git_vectors` lazily and so we can't
reliably bsearch the index and tree content without applying a
`git_vector_sort()` first.
This also fixes some missed places where size_t can be used and
where const can be applied to a couple internal functions.
This makes the diff functions that take callbacks both take
the payload parameter after the callback function pointers and
pass the payload as the last argument to the callback function
instead of the first. This should make them consistent with
other callbacks across the API.
3f9eb1e introduced support for SSL certificates issued for IP
addresses, making use of in_addr and in_addr6 structs. On FreeBSD
these are defined in (a file included in) <netinet/in.h>, so include
that file on FreeBSD and get the build working again.
The workdir iterator has always tried to ignore .git files, but
it turns out there were some bugs. This makes it more robust at
ignoring .git files.
This also makes iterators always check ".git" case insensitively
regardless of the properties of the system. This will make libgit2
skip ".GIT" and the like. This is different from core git, but on
systems with case insensitive but case preserving file systems,
allowing ".GIT" to be added is problematic.
This checks for a leading '.' before looking for the invalid
tree entry names. Even on pretty high levels of optimization,
this seems to make a measurable improvement.
I accidentally used && in the check initially instead of || and
while debugging ended up improving the error reporting of issues
with adding tree entries. I thought I'd leave those changes, too.
A number of diff APIs and the `git_checkout_index` API take a
`git_repository` object an operate on the index. This updates
them to take a `git_index` pointer explicitly and only fall back
on the `git_repository` index if the index input is NULL. This
makes it easier to operate on a temporary index.
The index iterator could previously only be created from a repo
object, but this allows creating an iterator from a `git_index`
object instead (while keeping, though renaming, the old function).
The existing p_lstat implementation on win32 is not quite POSIX
compliant when setting errno to ENOTDIR. This adds an option to
make is be compliant so that code (such as checkout) that cares
to have separate behavior for ENOTDIR can use it portably.
This also contains a couple of other minor cleanups in the
posix_w32.c implementations to avoid unnecessary work.
Using the builtin strcmp and strcasecmp as function pointers is
problematic on win32. This adds internal implementations and
divorces us from the platform linkage.
Returning NULL for the string when we haven't signaled an error
condition is counter-intuitive and causes unnecessary edge
cases. Return an empty string when asking for a string value for a
configuration variable such as '[section] var' to avoid these edge
cases.
If the distinction between no value and an empty value is needed, this
can be retrieved from the entry directly. As a side-effect, this
change stops the int parsing functions from segfaulting on such a
variable.
This fixes a number of warnings and problems with cross-platform
builds. Among other things, it's not safe to name a member of a
structure "strcmp" because that may be #defined.
This is a major reworking of checkout strategy options. The
checkout code is now sensitive to the contents of the HEAD tree
and the new options allow you to update the working tree so that
it will match the index content only when it previously matched
the contents of the HEAD. This allows you to, for example, to
distinguish between removing files that are in the HEAD but not
in the index, vs just removing all untracked files.
Because of various corner cases that arise, etc., this required
some additional capabilities in rmdir and other utility functions.
This includes the beginnings of an implementation of code to read
a partial tree into the index based on a pathspec, but that is
not enabled because of the possibility of creating conflicting
index entries.
There are some diff functions that are useful in a rewritten
checkout and this lays some groundwork for that. This contains
three main things:
1. Share the function diff uses to calculate the OID for a file
in the working directory (now named `git_diff__oid_for_file`
2. Add a `git_diff__paired_foreach` function to iterator over
two diff lists concurrently. Convert status to use it.
3. Move all the string/prefix/index entry comparisons into
function pointers inside the `git_diff_list` object so they
can be switched between case sensitive and insensitive
versions. This makes them easier to reuse in various
functions without replicating logic. As part of this, move
a couple of index functions out of diff.c and into index.c.
Diff uses a `git_strarray` of path specs to represent a subset
of all files to be processed. It is useful to be able to reuse
this filtering in other places outside diff, so I've moved it
into a standalone set of utilities.
This makes it so that the check if a file is ignored will be
deferred until requested on the workdir iterator, instead of
aggressively evaluating the ignore rules for each entry. This
should improve performance because there will be no need to
check ignore rules for files that are already in the index.
So, @nulltoken created a failing test case for checkout that
proved to be particularly daunting. If checkout is given only
a very limited strategy mask (e.g. just GIT_CHECKOUT_CREATE_MISSING)
then it is possible for typechange/rename modifications to leave it
unable to complete the request. That's okay, but the existing code
did not have enough information not to generate an error (at least
for tree/blob conflicts).
This led me to a significant reorganization of the code to handle
the failing case, but it has three benefits:
1. The test case is handled correctly (I think)
2. The new code should actually be much faster than the old code
since I decided to make checkout aware of diff list internals.
3. The progress value accuracy is hugely increased since I added
a fourth pass which calculates exactly what work needs to be
done before doing anything.
* Rework GIT_DIRREMOVAL values to GIT_RMDIR flags, allowing
combinations of flags
* Add GIT_RMDIR_EMPTY_PARENTS flag to remove parent dirs that
are left empty after removal
* Add GIT_MKDIR_VERIFY_DIR to give an error if item is a file,
not a dir (previously an EEXISTS error was ignored, even for
files) and enable this flag for git_futils_mkpath2file call
* Improve accuracy of error messages from git_futils_mkdir
This fix makes libgit2 capable of parsing annotated tag objects that lack
the optional message/description field.
Previously, libgit2 treated this field as mandatory and raised a tag_error on
such tags. However, the message field is optional.
An example of such a tag is refs/tags/v2.6.16.31-rc1 in Linux:
$ git cat-file tag refs/tags/v2.6.16.31-rc1
object afaa018cefb6af63befef1df7d8febaae904434f
type commit
tag v2.6.16.31-rc1
tagger Adrian Bunk <bunk@stusta.de> 1162716505 +0100
$
This improves docs in some of the public header files, cleans
up and improves some of the example code, and fixes a couple
of pedantic warnings in places.
This adds a new API that allows users to reload the config if the
file has changed on disk. A new config callback function to
refresh the config was added.
The modified time and file size are used to test if the file needs
to be reloaded (and are now stored in the disk backend object).
In writing tests, just using mtime was a problem / race, so I
wanted to check file size as well. To support that, I extended
`git_futils_readbuffer_updated` to optionally check file size in
addition to mtime, and I added a new function `git_filebuf_stats`
to fetch the mtime and size for an open filebuf (so that the
config could be easily refreshed after a write).
Lastly, I moved some similar file checking code for attributes
into filebuf. It is still only being used for attrs, but it
seems potentially reusable, so I thought I'd move it over.
This improves the naming for the rename related functionality
moving it to be called `git_diff_find_similar()` and renaming
all the associated constants, etc. to make more sense.
I also moved the new code (plus the existing `git_diff_merge`)
into a new file `diff_tform.c` where I can put new functions
related to manipulating git diff lists.
This also updates the implementation significantly from the
last revision fixing some ordering issues (where break-rewrite
needs to be handled prior to copy and rename detection) and
improving config option handling.
This adds a `git_diff_patch_print()` API which is more like the
existing API to "print" a patch from an entire `git_diff_list`
but operates on a single `git_diff_patch` object.
Also, it rewrites the `git_diff_patch_to_str()` API to use that
function (making it very small).
This implements the basis for diff rename and copy detection,
although it is based on simple SHA comparison right now instead
of using a matching algortihm. Just as `git_diff_merge` can be
used as a post-pass on diffs to emulate certain command line
behaviors, there is a new API `git_diff_detect` which will
update a diff list in-place, adjusting some deltas to RENAMED
or COPIED state (and also, eventually, splitting MODIFIED deltas
where the change is too large into DELETED/ADDED pairs).
This also adds a new test repo that will hold rename/copy/split
scenarios. Right now, it just has exact-match rename and copy,
but the tests are written to use tree diffs, so we should be able
to add new test scenarios easily without breaking tests.
libcryto's SHA-1 implementation is measurably better than the one that
ships with the library. If we link to it for HTTPS support already,
use that implementation instead.
Testing on a ~600MB of the linux repository, this reduces indexing
time by 40% and removes the hashing from the top spot in the perf
output.
Added `struct git_config_entry`: a git_config_entry contains the key, the value, and the config file level from which a config element was found.
Added `git_config_open_level`: build a single-level focused config object from a multi-level one.
We are now storing `git_config_entry`s in the khash of the config_file
- make sure temporary streamed blobs are created under the
.git/objects folder and not in the current path, whatever it is.
- do not make the name of the temp file depend on the hintpath.
git_index_read_tree() was exposing a parameter to provide the user with
a progress indicator. Unfortunately, due to the recursive nature of the
tree walk, the maximum number of items to process was unknown. Thus,
the indicator was only counting processed entries, without providing
any information how the number of remaining items.
The new Win32 global path search was not working with the
environment variable tests. But when I fixed the test, the new
codes use of getenv() was causing more failures (presumably because
of caching on Windows ???). This fixes the global file lookup to
always go directly to the Win32 API in a predictable way.
Introduce git_remote_stop() which sets a variable that is checked by
the fetch process in a few key places. If this is variable is set, the
fetch is aborted.
Fixed no-submodule speedup of new checkout code. Fixed missing
final update to progress (which may go away, I realize). Fixed
unused structure in header and incorrect comment.
To answer if a single given file should be ignored, the path to
that file has to be processed progressively checking that there
are no intermediate ignored directories in getting to the file
in question. This enables that, fixing the broken old behavior,
and adds tests to exercise various ignore situations.
Because fnmatch uses recursion, there were some input sequences
that cause seriously degenerate behavior. This imports a fix
that imposes a max recursion limiter to avoid the worst of it.
We used to require loose references to contain only an OID (possibly
after trimming the string). This is however not enough for letting us
lookup FETCH_HEAD, which can have a lot of content after the initial
OID.
Change the parsing rules so that a loose refernce must e at least 40
bytes long and the 41st (if it's there) must be accepted by
isspace(3). This makes the trim unnecessary, so only do it for
symrefs. This fixes#977.
The fix for fetching from empty repositories (22935b06d protocol:
don't store flushes; 2012-10-07) forgot to take into account the
deletion of the flush pkt in the HTTP transport. As a result, the HEAD
ref advertisement where we detect the remote's capabilities was
deleted instead. Fix this.
This started as a complex new test for checkout going through the
"typechanges" test repository, but that revealed numerous issues
with checkout, including:
* complete failure with submodules
* failure to create blobs with exec bits
* problems when replacing a tree with a blob because the tree
"example/" sorts after the blob "example" so the delete was
being processed after the single file blob was created
This fixes most of those problems and includes a number of other
minor changes that made it easier to do that, including improving
the TYPECHANGE support in diff/status, etc.
This is just some cleanup code, rearranging some of the checkout
code where TYPECHANGE support was added and adding some comments
to the diff header regarding the constants.
When I wrote the diff code, I based it on core git's diff output
which tends to split a type change into an add and a delete. But
core git's status has the notion of a T (typechange) flag for a
file. This introduces that into our status APIs and modifies the
diff code so it can be forced to not split type changes.
The adds a test for the submodule diff capabilities and then
fixes a few bugs with how the output is generated. It improves
the accuracy of OIDs in the diff delta object and makes the
submodule output more closely mirror the OIDs that will be used
by core git.
There are a few cases where diff should leave directories in
the diff list if we want to match core git, such as when the
directory contains a .git dir. That feature was lost when I
introduced some of the new submodule handling.
This restores that and then fixes a couple of related to diff
output that are triggered by having diffs with directories in
them.
Also, this adds a new flag that can be passed to diff if you
want diff output to actually include the file content of any
untracked files.
The reference is only needed inside the function. We mistakenly
increased the reference counter causing the ODB not to get freed and
leaking descriptors.
Storing flushes in the refs vector doesn't let us recognize when the
remote is empty, as we'd always introduce at least one element into
it. These flushes aren't necessary, so we can simply ignore them.
We don't have anything useful that we could do with that oid anyway (We
need to query the submodule for the HEAD commit instead).
Without this, the following code creates the error "Failed to read
descriptor: Is a directory" when run against the submod2 test-case:
const char* oidstr = "873585b94bdeabccea991ea5e3ec1a277895b698";
git_tree* tree = resolve_commit_oid_to_tree(g_repo, oidstr);
git_diff_list* diff = NULL;
cl_assert(tree);
cl_git_pass(git_diff_workdir_to_tree(g_repo, NULL, tree, &diff));
1. teach diff.c:maybe_modified to query git_submodule_status for the
modification state of a submodule. According to the
git_submodule_status docs, it will filter for to-ignore states
already.
2. teach diff_output.c:get_workdir_content to check the submodule status
again and create a line like:
Subproject commit <SHA-1>\n
or
Subproject comimt <SHA-1>-dirty\n
like git.git does.
diff_output.c:get_blob_content used to try to read the submodule commit
as a blob in the superproject's odb. Of course it cannot find it and
errors out with GIT_ENOTFOUND, implcitly terminating the whole diff
output.
This patch teaches it to create a text that describes the submodule
instead. The text looks like:
Subproject commit <SHA1>\n
which is what git.git does, too.
Together with include-tag, this make us behave more like git. After a
fetch, try to create any tags the remote told us about for which we
have objects locally.
Indicate whether the error comes from the ref already existing or
elsewhere. We always perform the check and this lets the user write
more concise code.
There are a lot of places where the diff API gives the user access
to internal data structures and many of these were being exposed
through non-const pointers. This replaces them all with const
pointers for any object that the user can access but is still
owned internally to the git_diff_list or git_diff_patch objects.
This will probably break some bindings... Sorry!
This fixes all the bugs in the new diff patch code. The only
really interesting one is that when we merge two diffs, we now
have to actually exclude diff delta records that are not supposed
to be tracked, as opposed to before where they could be included
because they would be skipped silently by `git_diff_foreach()`.
Other than that, there are just minor errors.
Replacing the `git_iterator` object, this creates a simple API
for accessing the "patch" for any file pair in a diff list and
then gives indexed access to the hunks in the patch and the lines
in the hunk. This is the initial implementation of this revised
API - it is still broken, but at least builds cleanly.
This file is not just read if the global config file (%HOME%/.gitconfig)
is not found, however, it is used everytime but with lower priority.
Signed-off-by: Sven Strickroth <email@cs-ware.de>
Do not hardcode the installation path of msysgit, but read installation path from registry.
Also "%PROGRAMFILES%\Git\etc" won't work on x64 systems with 64-bit libgit2, because
msysgit is x86 only and located in "%ProgramFiles(x86)%\Git\etc".
Signed-off-by: Sven Strickroth <email@cs-ware.de>
On most systems %USERPROFILE% is the same as %HOMEDRIVE%\%HOMEPATH%,
however, for windows machines in an AD or domain environment this
might be different and %HOMEDRIVE%\%HOMEPATH% seems to be better.
Signed-off-by: Sven Strickroth <email@cs-ware.de>
Use %HOME% before trying to figure out the windows user directory.
Users might set this as they are used on *nix systems.
Signed-off-by: Sven Strickroth <email@cs-ware.de>
Since quite a while now, git_branch_foreach has learnt to list branches
without the 'refs/heads/' or 'refs/remotes' prefixes.
This patch teaches git_tag_list to do the same for listing tags.
There has been discussion for a while about making some set of
the `giterr_set` type functions part of the public API for code
that is implementing new backends to libgit2. This makes the
`giterr_set_str()` and `giterr_set_oom()` functions public.
The old method was avoiding re-loading of packfiles by watching the mtime of the
pack directory. This causes the ODB to become stale if the directory and packfile
are written within the same clock millisecond, as when cloning a fairly small
repo.
This method tries to find the object in the cached packs, and forces a refresh when
that fails. This will cause extra stat'ing on a miss, but speeds up the success
case and avoids this race condition.
last_found is the last packfile a wanted object was found in. Since
last_found is shared among all searching threads, it might changes while
we're searching. As suggested by @arrbee, put a copy on the stack to fix
the race condition.
Defining the BOM as a string makes the array include the
NUL-terminator, which means that the memcpy is going to check for that
as well and thus never match for a nonempty file.
Define the array as three chars, which makes the size correct.
Wondows has its own HTTP library. Use that one when possible instead of
our own.
As we don't depend on them anymore, remove the http-parser library from
the Windows build, as well as the search for OpenSSL.
There is a bug in building the linked list of line records in the
diff iterator and also an off by one element error in the hunk
counts. This fixes both of these, adds some test data with more
complex sets of hunk and line diffs to exercise this code better.
This reduces the rate of syscalls for the common case of sequences of
object reads from the same pack.
Best of 5 timings for libgit2_clar before this patch:
real 0m5.375s
user 0m0.392s
sys 0m3.564s
After applying this patch:
real 0m5.285s
user 0m0.356s
sys 0m3.544s
0.6% improvement in system time.
9.2% improvement in user time.
1.7% improvement in elapsed time.
Confirmed a 0.6% reduction in number of system calls with strace.
Expect greater improvement for graph-traversal with large packs.
Fixed some minor `git_repository_hashfile` issues:
- Fixed incorrect doc (saying that repo could be NULL)
- Added checking of object type value to acceptable ones
- Added more tests for various parameter permutations
The existing `git_odb_hashfile` does not apply text filtering
rules because it doesn't have a repository context to evaluate
the correct rules to apply. This adds a new hashfile function
that will apply repository-specific filters (based on config,
attributes, and filename) before calculating the hash.
In the process of adding tests for the max file size threshold
(which treats files over a certain size as binary) there seem to
be a number of problems in the new code with detecting binaries.
This should fix those up, as well as add a test for the file
size threshold stuff.
Also, this un-deprecates `GIT_DIFF_LINE_ADD_EOFNL`, since I
finally found a legitimate situation where it would be returned.
Example: a cached node is owned only by the cache (refcount == 1).
Thread A holds the lock and determines that the entry which should get
cached equals the node (git_oid_cmp(&node->oid, &entry->oid) == 0).
It frees the given entry to instead return the cached node to the user
(entry = node). Now, before Thread A happens to increment the refcount
of the node *outside* the cache lock, Thread B tries to store another
entry and hits the slot of the node before, decrements its refcount and
frees it *before* Thread A gets a chance to increment for the user.
git_cached_obj_incref(entry);
git_mutex_lock(&cache->lock);
{
git_cached_obj *node = cache->nodes[hash & cache->size_mask];
if (node == NULL) {
cache->nodes[hash & cache->size_mask] = entry;
} else if (git_oid_cmp(&node->oid, &entry->oid) == 0) {
git_cached_obj_decref(entry, cache->free_obj);
entry = node;
} else {
git_cached_obj_decref(node, cache->free_obj);
// Thread B is here
cache->nodes[hash & cache->size_mask] = entry;
}
}
git_mutex_unlock(&cache->lock);
// Thread A is here
/* increase the refcount again, because we are
* returning it to the user */
git_cached_obj_incref(entry);
Often `git_odb_read_header` will "fail" and have to read the
entire object into memory instead of just the header. When this
happens, the object is loaded and then disposed of immediately,
which makes it difficult to efficiently use the header information
to decide if the object should be loaded (since attempting to do
so will often result in loading the object twice).
This commit takes the existing code and reorganizes it to have
two new functions:
- `git_odb__read_header_or_object` which acts just like the old
read header function except that it returns the object, too, if
it was forced to load the whole thing. It then becomes the
callers responsibility to free the `git_odb_object`.
- `git_object__from_odb_object` which was extracted from the old
`git_object_lookup` and creates a subclass of `git_object` from
an existing `git_odb_object` (separating the ODB lookup from the
`git_object` creation). This allows you to use the first header
reading function efficiently without instantiating the
`git_odb_object` twice.
There is no net change to the behavior of any of the existing
functions, but this allows internal code to tap into the ODB
lookup and object creation to be more efficient.
This commit adds a max_size value in the public `git_diff_options`
structure so that the user can automatically flag blobs over a
certain size as binary regardless of other properties.
Also, and perhaps more importantly, this moves binary detection
to be as early as possible in the diff traversal inner loop and
makes sure that we stop loading objects as soon as we decide that
they are binary.
The `git_diff_iterator_num_files` API was problematic, since we
don't actually know the exact number of files to be iterated over
until we load those files into memory. This replaces it with a
new `git_diff_iterator_progress` API that goes from 0 to 1, and
moves and renamed the old API for the internal places that can
tolerate a max value instead of an exact value.
Previously when diffing blobs, the diff code just ran with a NULL
repository object. Of course, that's not necessary and the test
for a NULL repo was confusing. This makes the blob diff run with
the repo that contains the blobs and clarifies the test that it
is possible to be diffing data where the path is unknown.
This adds support to diff and status for running filters (a la crlf)
on blobs in the workdir before computing SHAs and before generating
text diffs. This ended up being a bit more code change than I had
thought since I had to reorganize some of the diff logic to minimize
peak memory use when filtering blobs in a diff.
This also adds a cap on the maximum size of data that will be loaded
to diff. I set it at 512Mb which should match core git. Right now
it is a #define in src/diff.h but it could be moved into the public
API if desired.
This refactors the diff output code so that an iterator object
can be used to traverse and generate the diffs, instead of just
the `foreach()` style with callbacks. The code has been rearranged
so that the two styles can still share most functions.
This also replaces `GIT_REVWALKOVER` with `GIT_ITEROVER` and uses
that as a common error code for marking the end of iteration when
using a iterator style of object.
SSL_get_error() allows to receive a result code for various SSL
operations. Depending on the return value (see man (3) SSL_get_error)
there might be additional information in the OpenSSL error queue. Return
the queued message if available, otherwise set an error message
corresponding to the return code.
There is a better and less fragile way to calculate time offsets. Let
the OS take care of dealing with DST and simply take the the offset
between the local time and UTC that it gives us.
Passing SSL_VERIFY_PEER makes OpenSSL shut down the connection if the
certificate is invalid, without giving us a chance to ignore that
error. Pass SSL_VERIFY_NONE and call SSL_get_verify_result if the user
wanted us to check.
When no CNs match, we used to jump to on_error which gave a bogus
error as that's for OpenSSL errors. Jump to cert_fail so we tell the
user that the error came from checking the certificate.
This expands the types of peeling that `git_object_peel` knows
how to do to include TAG -> BLOB peeling, and makes the errors
slightly more consistent depending on the situation. It also
adds a new special behavior where peeling to ANY will peel until
the object type changes (e.g. chases TAGs to a non-TAG).
Using this expanded peeling, this replaces peeling code that was
embedded in `git_tag_peel` and `git_reset`.
