String mememory is now managed in a much more sane manner.
Fixes include:
- git_person email and name is no longer limited to 64 characters
- git_tree_entry filename is no longer limited to 255 characters
- raw objects are properly opened & closed the minimum amount of
times required for parsing
- unit tests no longer leak
- removed 5 other misc memory leaks as reported by Valgrind
- tree writeback no longer segfaults on rare ocassions
The git_person struct is no longer public. It is now managed by the
library, and getter methods are in place to access its internal
attributes.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Since commit 70aab459, the msvc and MinGW builds have relied on
the built-in implementation of ntohl() and htonl(), rather than
linking the wsock32 library. The new index manipulation code now
calls ntohs()/htons() in addition to ntohl()/htonl(), so we need
to provide a built-in implementation of the 16-bit functions.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
The tree array wasn't being initialized when instantiating a tree object
in memory instead of loading it from disk.
New unit tests added to check for the problem.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Tag files can now be created and modified in-memory (all the setter
methods have been implemented), and written back to disk using the
generic git_object_write() method.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
git_tree_entry_byname was dereferencing a NULL pointer when the searched
file couldn't be found on the tree.
New test cases have been added to check for entry access methods.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
All the setter methods for git_tree have been added, including the
setters for attributes on each git_tree_entry and methods to add/remove
entries of the tree.
Modified trees and trees created in-memory from scratch can be written
back to the repository using git_object_write().
Signed-off-by: Vicent Marti <tanoku@gmail.com>
All repository objects can now be created from scratch in memory using
either the git_object_new() method, or the corresponding git_XXX_new()
for each object.
So far, only git_commits can be written back to disk once created in
memory.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
All the required git_commit_set_XXX methods have been implemented; all
the attributes of a commit object can now be modified in-memory.
The new method git_object_write() automatically writes back the
in-memory changes of any object to the repository. So far it only
supports git_commit objects.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
The new 'git__source_printf' does an overflow-safe printf on a source
bfufer.
The new 'git__source_write' does an overflow-safe byte write on a source
buffer.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
The 'git_obj' structure is now called 'git_rawobj', since
it represents a raw object read from the ODB.
The 'git_repository_object' structure is now called 'git_object',
since it's the base object class for all objects.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
git_repository_object has now several internal methods to write back the
object information in the repository.
- git_repository__dbo_prepare_write()
Prepares the DBO object to be modified
- git_repository__dbo_write()
Writes new bytes to the DBO object
- git_repository__dbo_writeback()
Writes back the changes to the repository
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Added several methods to access:
- The ODB behind a repo
- The SHA1 id behind a generic repo object
- The type of a generic repo object
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Some compilers give linking problems when exporting 'uint32_t' as a
return type in the external API. Use generic types instead.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
A new method 'git_repository_object_free' allows to manually force the
freeing of a repository object, even though they are still automatically
managed by the repository and don't need to be freed by the user.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
The interface for loading and parsing tree objects from a repository has
been completed with all the required accesor methods for attributes,
support for manipulating individual tree entries and a new unit test
t0901-readtree which tries to load and parse a tree object from a
repository.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
The new 'git_index' structure is an in-memory representation
of a git index on disk; the 'git_index_entry' structures represent
each one of the file entries on the index.
The following calls for index instantiation have been added:
git_index_alloc(): instantiate a new index structure
git_index_free(): free an existing index
git_index_clear(): clear all the entires in an existing file
The following calls for index reading and writing have been added:
git_index_read(): update the contents of the index structure from
its file on disk.
Internally implemented through:
git_index__parse()
Index files are stored on disk in network byte order; all integer fields
inside them are properly converted to the machine's byte order when
loading them in memory. The parsing engine also distinguishes
between normal index entries and extended entries with 2 extra bytes
of flags.
The 'TREE' extension for index entries is also loaded into memory:
Tree caches stored in Index files are loaded into the
'git_index_tree' structure pointed by the 'tree' pointer inside
'git_index'.
'index->tree' points to the root node of the tree cache; the full tree
can be traversed through each of the node's 'tree->children'.
Index files can be written back to disk through:
git_index_write(): atomic writing of existing index objects
backed by internal method git_index__write()
The following calls for entry manipulation have been added:
git_index_add(): insert an empty entry to the index
git_index_find(): search an entry by its path name
git_index__append(): appends a new index entry to the end of the
list, resizing the entries array if required
New index entries are always inserted at the end of the array; since the
index entries must be sorted for it to be internally consistent, the
index object is only sorted once, and if required, before accessing the
whole entriea array (e.g. before writing to disk, before traversing,
etc).
git_index__remove_pos(): remove an index entry in a specific position
git_index__sort(): sort the entries in the array by path name
The entries array is sorted stably and in place using an
insertion sort, which ought to be the most efficient approach
since the entries array is always mostly-sorted.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
The struct 'git_filelock' represents an atomically-locked
file, git-style.
Locked files can be modified atomically through the new file lock
interface:
int git_filelock_init(git_filelock *lock, const char *path);
int git_filelock_lock(git_filelock *lock, int append);
void git_filelock_unlock(git_filelock *lock);
int git_filelock_commit(git_filelock *lock);
int git_filelock_write(git_filelock *lock, const char *buffer, size_t length);
Signed-off-by: Vicent Marti <tanoku@gmail.com>
The old 'git_revpool' object has been removed and
split into two distinct objects with separate
functionality, in order to have separate methods for
object management and object walking.
* A new object 'git_repository' does the high-level
management of a repository's objects (commits, trees,
tags, etc) on top of a 'git_odb'.
Eventually, it will also manage other repository
attributes (e.g. tag resolution, references, etc).
See: src/git/repository.h
* A new external method
'git_repository_lookup(repo, oid, type)'
has been added to the 'git_repository' API.
All object lookups (git_XXX_lookup()) are now
wrappers to this method, and duplicated code
has been removed. The method does automatic type
checking and returns a generic 'git_revpool_object'
that can be cast to any specific object.
See: src/git/repository.h
* The external methods for object parsing of repository
objects (git_XXX_parse()) have been removed.
Loading objects from the repository is now managed
through the 'lookup' functions. These objects are
loaded with minimal information, and the relevant
parsing is done automatically when the user requests
any of the parsed attributes through accessor methods.
An attribute has been added to 'git_repository' in
order to force the parsing of all the repository objects
immediately after lookup.
See: src/git/commit.h
See: src/git/tag.h
See: src/git/tree.h
* The previous walking functionality of the revpool
is now found in 'git_revwalk', which does the actual
revision walking on a repository; the attributes
when walking through commits in a database have been
decoupled from the actual commit objects.
This increases performance when accessing commits
during the walk and allows to have several
'git_revwalk' instances working at the same time on
top of the same repository, without having to load
commits in memory several times.
See: src/git/revwalk.h
* The old 'git_revpool_table' has been renamed to
'git_hashtable' and now works as a generic hashtable
with support for any kind of object and custom hash
functions.
See: src/hashtable.h
* All the relevant unit tests have been updated, renamed
and grouped accordingly.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Tag objects are now properly loaded from the revision pool.
New test t0801 checks for loading a parsing a series of tags, including
the tag of a tag.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
The 'parse_oid' and 'parse_person' methods which were used by the commit
parser are now global so they can be used when parsing other objects.
The 'git_commit_person' struct has been changed to a generic
'git_person'.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Packed objects inside packfiles are now properly unpacked when calling
the git_odb__read_packed() method; delta'ed objects are also properly
generated when needed.
A new unit test 0204-readpack tries to read a couple hundred packed
objects from a standard packed repository.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
The basic information (pointed trees and blobs) of each tree object in a
revision pool can now be parsed and queried.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
The following new external methods have been added:
GIT_EXTERN(const char *) git_commit_message_short(git_commit *commit);
GIT_EXTERN(const char *) git_commit_message(git_commit *commit);
GIT_EXTERN(time_t) git_commit_time(git_commit *commit);
GIT_EXTERN(const git_commit_person *) git_commit_committer(git_commit *commit);
GIT_EXTERN(const git_commit_person *) git_commit_author(git_commit *commit);
GIT_EXTERN(const git_tree *) git_commit_tree(git_commit *commit);
A new structure, git_commit_person has been added to represent a
commit's author or committer.
The parsing of a commit has been split in two phases.
When adding a commit to the revision pool:
- the commit's ODB object is opened
- its raw contents are parsed for commit TIME, PARENTS and TREE
(the minimal amount of data required to traverse the pool)
- the commit's ODB object is closed
When querying for extended information on a commit:
- the commit's ODB object is reopened
- its raw contents are parsed for the requested information
- the commit's ODB object remains open to handle additional queries
New unit tests have been added for the new functionality:
In t0401-parse: parse_person_test
In t0402-details: query_details_test
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Commits now store pointers to their tree objects.
Tree objects now work as separate git_revpool_object
entities.
Tree objects can be loaded and parsed inedependently
from commits.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
git_revpool_object now has a type identifier for each object
type in a revpool (commits, trees, blobs, etc).
Trees can now be stored in the revision pool.
git_revpool_tableit now supports filtering objects by their
type when iterating through the object table.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Created commit objects in t0401-parse weren't being freed properly.
Updated the API documentation to note that commit objects are owned
by the revision pool and should not be freed manually.
The parents list of each commit was being freed twice after each test.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Previously the objects table was being freed, but not
the actuall commits. All git_commit objects are freed
and hence invalidated when freeing the git_rp object
they belong to.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
This fix had been delayed by Ramsay because on 32-bit systems it
highlights the fact that off_t is set to an invalid value.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
This reduces the global namespace pollution. These functions
were the only remaining external symbols (with the exception
of an PPC_SHA1 build) which did not start with 'git', and
since these are private library symbols the 'git__' prefix is
appropriate.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Given that the sha1.h header file should never be included into
any other file, since it represents an implementation detail of
hash.c, we remove the header and inline it's content.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
On Intel machines, the msvc compiler defines the CPU architecture
macros _M_IX86 and _M_X64 (equivalent to __i386__ and __x86_64__
respectively). Use these macros in the pre-processor expression
to select the "fast" definition of the {get,put}_be32() macros.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
When git_oid_to_string() was passed a buffer size larger than
GIT_OID_HEXSZ+1, the function placed the c-string NUL char at
the wrong position. Fix the code to place the NUL at the end
of the (possibly truncated) oid string.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
In order to avoid inconsistent definitions of type off_t, all
compilation units should include the "common.h" header file
before certain system headers (those which directly or indirectly
lead to the definition of off_t). The "common.h" header contains
the definition of _FILE_OFFSET_BITS to select 64-bit file offsets.
The symptom of this inconsistency, while compiling with -Wextra, is
the following warning:
In file included from src/common.h:50,
from src/commit.c:28:
src/util.h: In function git__is_sizet:
src/util.h:41: warning: comparison between signed and unsigned
In order to fix the problem, we simply remove the #include <time.h>
statement at the head of src/commit.c. Note that src/commit.h also
includes <time.h>.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Andreas Ericsson <ae@op5.se>
gcc (4.4.0) issues the following warning:
src/revobject.c:33: warning: dereferencing type-punned pointer \
will break strict-aliasing rules
We suppress the warning by copying the first 4 bytes from the oid
structure into an 'unsigned int' using memcpy(). This will also
fix any potential alignment issues on certain platforms.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Andreas Ericsson <ae@op5.se>
In particular, doxygen issues the following warning:
.../src/git/revwalk.h:86: Warning: The following parameters of \
gitrp_sorting(git_revpool *pool, unsigned int sort_mode) are \
not documented:
parameter 'pool'
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Andreas Ericsson <ae@op5.se>
In particular, sparse issues the following warnings:
src/revobject.c:29:14: warning: symbol 'max_load_factor' was \
not declared. Should it be static?
src/revobject.c:31:14: warning: symbol 'git_revpool_table__hash' was \
not declared. Should it be static?
In order to suppress these warnings, we simply declare them as
static, since they are not (currently) referenced outside of this
file.
In the case of max_load_factor, this is probably correct. However,
this may not be appropriate for git_revpool_table__hash(), given
how it is named. So, this should either be re-named to reflect it's
non-external status, or a declaration needs to be added to the
revobject.h header file.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Andreas Ericsson <ae@op5.se>
In order to suppress this warning, we could simply replace the
constant 0 with NULL. However, in this case, replacing the
comparison with 0 by !buffer is more idiomatic.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Andreas Ericsson <ae@op5.se>
In particular, the compiler issues the following warning:
src/revwalk.c(61) : warning C4244: '=' : conversion from \
'unsigned int' to 'unsigned char', possible loss of data
In order to suppress the warning, we change the type of the
sorting "enum" field of the git_revpool structure to be consistent
with the sort_mode parameter of the gitrp_sorting() function.
Note that if the size of the git_revpool structure is an issue,
then we could change the type of the sort_mode parameter instead.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Andreas Ericsson <ae@op5.se>
In particular, the compiler issues the following warnings:
src/revobject.c(29) : warning C4305: 'initializing' : truncation \
from 'double' to 'const float'
src/revobject.c(56) : warning C4244: '=' : conversion from \
'const float' to 'unsigned int', possible loss of data
src/revobject.c(149) : warning C4244: '=' : conversion from \
'const float' to 'unsigned int', possible loss of data
In order to suppress the warnings we change the type of max_load_factor
to double, rather than change the initialiser to 0.65f, and cast the
result type of the expressions to 'unsigned int' as expected by the
assignment operators. Note that double should be able to represent all
unsigned int values without loss.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Andreas Ericsson <ae@op5.se>
These warnings are issued by both gcc (-Wextra) and msvc (-W3).
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Andreas Ericsson <ae@op5.se>
On the msvc build, the tests t0401-parse and t0501-walk both
crash with a runtime error (ACCESS_VIOLATION). This is caused
by writing to un-allocated memory due to an under-allocation
of a git_revpool_table data structure.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Andreas Ericsson <ae@op5.se>
sorting ('prev' pointers in the linked list are no longer lost).
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
The GIT_RPSORT_XXX flags have been moved to the external API,
and a new method 'gitrp_sorting(...)' has been added to safely
change the sorting method of a revision pool.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
'git_commit_list_toposort()' and 'git_commit_list_timesort()' now
sort a commit list by topological and time order respectively.
Both sorts are stable and in place.
'git_commit_list_append' has been replaced by 'git_commit_list_push_back'
and 'git_commit_list_push_front'.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
Fixed issue when generating pending commits list during iteration.
The 'git_commit_lookup' function will now check the pool's cache
for commits which have been previously loaded/parsed; there can only
be a single 'git_commit' structure for each commit on the same pool.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
All the objects which will will be eventually transversable from
a revision pool (commits, trees, etc) now inherit from the
'git_revpool_object' structure which identifies them with their
own OID.
Furthermore, the 'git_revpool_table' and related functions have
been added, which allow for constant time lookup (hash table)
of the loaded revpool objects based on their OID.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
The 'gitrp_next()' method now correctly does a revision walking
of all the pushed revisions in arbritary ordering.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
git_commit_lookup() now creates commit references
without loading them from the ODB.
git_commit_parse() creates a commit reference, loads
it and parses it from the ODB.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
Basic support for iterating the revpool.
The following functions of the revwalk API have been partially
implemented:
void gitrp_reset(git_revpool *pool);
void gitrp_push(git_revpool *pool, git_commit *commit);
void gitrp_prepare_walk(git_revpool *pool);
git_commit *gitrp_next(git_revpool *pool);
Parsed commits' parents are now also parsed and stored in a
"git_commit_list" structure (linked list).
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
A few initial tests for commit parsing:
"parse_buffer_test" tests git_commit__parse_buffer() with
several malformed commit messages and a few corner cases
which should pass.
"parse_oid_test" tests git_commit__parse_oid() with several
malformed commit lines containing broken SHA1 OIDs.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
The external API function "git_commit_parse" has been renamed
to "git_commit_lookup" and has been partially implemented with
support for loading commits straight from the ODB. It still lacks
the functionality to lookup cached commits in the revpool and to
resolve tags to commits.
The following internal functions have been partially implemented:
int git_commit__parse_buffer(...);
int git_commit__parse_time(...);
int git_commit__parse_oid(...);
Commits are now fully parsed but the generated parent and tree
references are not handled yet.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
In particular, using the normal (or production) compiler
warning level (-W3), msvc complains as follows:
.../sha1.c(244) : warning C4018: '<' : signed/unsigned mismatch
.../sha1.c(270) : warning C4244: 'function' : conversion from \
'unsigned __int64' to 'unsigned long', possible loss of data
.../sha1.c(271) : warning C4244: 'function' : conversion from \
'unsigned __int64' to 'unsigned long', possible loss of data
Note that gcc issues a similar complaint about line 244 when
compiling with -Wextra.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Commit 5dddf7c (Add block-sha1 in favour of the mozilla routines
2010-04-14) introduced the "bswap.h" header file which contains
an inline function (default_swab32()). The msvc compiler does
not support the inline keyword which causes the build to fail
with a syntax error.
However, msvc does support inline functions using the __inline
keyword language extension. We already have the GIT_INLINE()
macro that allows us to hide this syntatic difference. In order
to fix the build, we simply use GIT_INLINE() in the definition
of the default_swab32() function.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
* ramsay/dev:
Add a pack index 'virtual function' to fetch an index entry
Add a pack index 'virtual function' to search by file offset
Change the interface of the pack index search function
Add an 64-bit offset table index bounds check for v2 pack index
Add a minimum size check when opening an v2 pack index file
win32: Add separate MinGW and MSVC compatability header files
Makefile: Add support for custom build options in config.mak file
Fix some coding style issues
Since block-sha1 from git.git has such excellent performance, we
can also get rid of the openssl dependency. It's rather simple
to add it back later as an optional extra, but we really needn't
bother to pull in the entire ssl library and have to deal with
linking issues now that we have the portable and, performance-wise,
truly excellent block-sha1 code to fall back on.
Since this requires a slight revamp of the build rules anyway, we
take the opportunity to fix including EXTRA_OBJS in the final build
as well.
The block-sha1 code was originally implemented for git.git by
Linus Torvalds <torvalds@linux-foundation.org> and was later
polished by Nicolas Pitre <nico@cam.org>.
Signed-off-by: Andreas Ericsson <ae@op5.se>
We don't use it yet, but now we have it there at least.
All the non-trivial parts of it appears to have been written
and contributed to git.git by some anonymous genius. The original
implementation was done by Paul Mackerras <paulus@samba.org>.
Signed-off-by: Andreas Ericsson <ae@op5.se>
Given an index entry number, the idx_get() function returns an
(version agnostic) index_entry structure containing all of the
information required to unpack the corresponding object from
the '.pack' file.
Since the v1 and v2 file formats differ in the layout of the
object records, we provide two implementations of the get
function and initialise the function pointer appropriately.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
In addition to searching the index by oid, we need to search by
'.pack' file offset, particularly when processing OBJ_OFS_DELTA
objects. Since the v1 and v2 file formats differ in the layout
of the object records, we provide two implementations of the
search function and initialise the (virtual) function pointer
appropriately.
Note that, as part of the creation of the 'offset index', we also
add a check that the offset data in the index is within the bounds
of the '.pack' file. Having sorted the file offsets, while creating
the index, we only need to check the smallest and largest values.
The offset index consists of the im_off_idx array, which contains
the index entry numbers sorted into file offset order, and the
im_off_next mapping array. The im_off_next array maps an index
entry number to the 'next' index entry in file offset order.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
In particular, on a successful search, we now return the index
entry number of the object rather than the '.pack' file offset.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
This reduces the global namespace pollution and allows for
a win32 compiler (eg. Open Watcom) to provide these routines
in a header other than <dirent.h> (eg in <io.h>).
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
