Temporary files when doing streaming writes are now stored inside the
Objects folder, to prevent issues when moving files between
disks/partitions.
Add support for block writes to the ODB again (for those backends that
cannot implement streaming).
Hey. Apologies in advance -- I broke your bindings.
This is a major commit that includes a long-overdue redesign of the
whole object-database structure. This is expected to be the last major
external API redesign of the library until the first non-alpha release.
Please get your bindings up to date with these changes. They will be
included in the next minor release. Sorry again!
Major features include:
- Real caching and refcounting on parsed objects
- Real caching and refcounting on objects read from the ODB
- Streaming writes & reads from the ODB
- Single-method writes for all object types
- The external API is now partially thread-safe
The speed increases are significant in all aspects, specially when
reading an object several times from the ODB (revwalking) and when
writing big objects to the ODB.
Here's a full changelog for the external API:
blob.h
------
- Remove `git_blob_new`
- Remove `git_blob_set_rawcontent`
- Remove `git_blob_set_rawcontent_fromfile`
- Rename `git_blob_writefile` -> `git_blob_create_fromfile`
- Change `git_blob_create_fromfile`:
The `path` argument is now relative to the repository's working dir
- Add `git_blob_create_frombuffer`
commit.h
--------
- Remove `git_commit_new`
- Remove `git_commit_add_parent`
- Remove `git_commit_set_message`
- Remove `git_commit_set_committer`
- Remove `git_commit_set_author`
- Remove `git_commit_set_tree`
- Add `git_commit_create`
- Add `git_commit_create_v`
- Add `git_commit_create_o`
- Add `git_commit_create_ov`
tag.h
-----
- Remove `git_tag_new`
- Remove `git_tag_set_target`
- Remove `git_tag_set_name`
- Remove `git_tag_set_tagger`
- Remove `git_tag_set_message`
- Add `git_tag_create`
- Add `git_tag_create_o`
tree.h
------
- Change `git_tree_entry_2object`:
New signature is `(git_object **object_out, git_repository *repo, git_tree_entry *entry)`
- Remove `git_tree_new`
- Remove `git_tree_add_entry`
- Remove `git_tree_remove_entry_byindex`
- Remove `git_tree_remove_entry_byname`
- Remove `git_tree_clearentries`
- Remove `git_tree_entry_set_id`
- Remove `git_tree_entry_set_name`
- Remove `git_tree_entry_set_attributes`
object.h
------------
- Remove `git_object_new
- Remove `git_object_write`
- Change `git_object_close`:
This method is now *mandatory*. Not closing an object causes a
memory leak.
odb.h
-----
- Remove type `git_rawobj`
- Remove `git_rawobj_close`
- Rename `git_rawobj_hash` -> `git_odb_hash`
- Change `git_odb_hash`:
New signature is `(git_oid *id, const void *data, size_t len, git_otype type)`
- Add type `git_odb_object`
- Add `git_odb_object_close`
- Change `git_odb_read`:
New signature is `(git_odb_object **out, git_odb *db, const git_oid *id)`
- Change `git_odb_read_header`:
New signature is `(size_t *len_p, git_otype *type_p, git_odb *db, const git_oid *id)`
- Remove `git_odb_write`
- Add `git_odb_open_wstream`
- Add `git_odb_open_rstream`
odb_backend.h
-------------
- Change type `git_odb_backend`:
New internal signatures are as follows
int (* read)(void **, size_t *, git_otype *, struct git_odb_backend *, const git_oid *)
int (* read_header)(size_t *, git_otype *, struct git_odb_backend *, const git_oid *)
int (* writestream)(struct git_odb_stream **, struct git_odb_backend *, size_t, git_otype)
int (* readstream)( struct git_odb_stream **, struct git_odb_backend *, const git_oid *)
- Add type `git_odb_stream`
- Add enum `git_odb_streammode`
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Clean up a provided absolute or relative directory path.
This prettification relies on basic operations such as coalescing multiple forward slashes into a single slash, removing '.' and './' current directory segments, and removing parent directory whenever '..' is encountered. If not empty, the returned path ends with a forward slash.
For instance, this will turn "d1/s1///s2/..//../s3" into "d1/s3/".
This only performs a string based analysis of the path. No checks are done to make sure the path actually makes sense from the file system perspective.
It is not a good idea to export these internal symbols now that they are
not required to run the unit tests.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
Some external functions were not being exported because they were using
the 'extern' keyword instead of the generic GIT_EXTERN() macro.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
The constructor to git_repository is now called
'git_repository_open(path)'
and takes a path to a git repository instead of an existing ODB object.
Unit tests have been updated accordingly and the two test repositories
have been merged into one.
Signed-off-by: Vicent Marti <tanoku@gmail.com>
No need to define inline as __inline because libgit2 code
should be using GIT_INLINE instead.
Signed-off-by: Julio Espinoza-Sokal <julioes@gmail.com>
Signed-off-by: Andreas Ericsson <ae@op5.se>
For information on FlushFileBuffers(), see the msdn document
at msdn.microsoft.com/en-us/library/aa364439(VS.85).aspx
Note that Windows 2000 is shown as the minimum windows version
to support FlushFileBuffers(), so if we wish to support Win9X
and NT4, we will need to add code to dynamically check if
kernel32.dll contains the function.
The only error return mentioned in the msdn document is
ERROR_INVALID_HANDLE, which is returned if the file/device
(eg console) is not buffered. The fsync(2) manpage says that
EINVAL is returned in errno, if "fd is bound to a special
file which does not support synchronization".
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
This test assumed that it was invoked in an empty directory,
which is true when run from the Makefile, and so would fail
if run standalone. In order to allow the test to work when
run from any directory, create a sub directory "dir-walk"
and chdir() into this directory while running the tests.
Also, add some additional tests.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
In particular, the git__mmap() and git__munmap() routines provide
the interface to platform specific memory-mapped file facilities.
We provide implementations for unix and win32, which can be found
in their own sub-directories.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
On windows, unless we use the O_BINARY flag in the open()
call, the file I/O routines will perform line ending
conversion (\r\n => \n on input, \n => \r\n on output).
In addition to the performance penalty, most files in the
object database are binary and will, therefore, become
corrupted by this conversion.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Win32 has a variant of mmap that is harder to use than POSIX, but
to run natively and efficiently on Win32 we need some form of it.
gitfo_map_ro() provides a basic mmap function for use in locations
where we need read-only random data access to large ranges of a file,
such as a pack-*.idx.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
When scanning the pack directory we need to see if the path
name is present for ".idx" when we discover a ".pack" file.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Our fileops API is currently private. We aren't planning on supplying
a cross-platform file API to applications that link to us. If we did,
we'd probably whole-sale publish fileops, not just the dirent code.
By moving it to be private we can also change the call signature to
permit the buffer to be passed down through the call chain. This is
very helpful when we are doing a recursive scan as we can reuse just
one buffer in all stack frames, reducing the impact the recursion has
on the stack frames in the data cache.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
This error code indicates the OS error code has a better value
describing the last error, as it is likely a network or local
file IO problem identified by a C library function call.
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
In particular, the gitfo_read_file() routine can be used to slurp
the complete file contents into an gitfo_buf structure. The buffer
content will be allocated by malloc() and may be released by the
gitfo_free_buf() routine. The io buffer type can be initialised
on the stack with the GITFO_BUF_INIT macro.
Signed-off-by: Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Since at least MS have something like GetFirstDirEnt() and
GetNextDirEnt() (presumably with superior performance), we
can let MS hackers add support for a dirent walker using
that API instead, while we stick with the posix-style
readdir() calls.
Signed-off-by: Andreas Ericsson <ae@op5.se>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
The idea is taken from Junio's work in read-cache.c, where
it's used for writing out the index without tap-dancing on
the poor harddrive. Since it's almost certainly useful for
cached writing of packfiles too, we turn it into a generic
API, making it perfectly simple to reuse it later.
gitfo_write_cached() has the same contract as gitfo_write(), it
returns GIT_SUCCESS if all bytes are successfully written (or were
at least buffered for later writing), and <0 if an error occurs
during buffer writing.
Signed-off-by: Andreas Ericsson <ae@op5.se>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
Since it doesn't make sense to make the disk access stuff
portable *AND* public (that's a job for each application
imo), we can take a shortcut and just support unixy stuff
for now and get away with coding most of it as macros.
Since we go with an internal API for starters and only
provide higher-level API's to the libgit users, we'll be
ok with this approach.
Signed-off-by: Andreas Ericsson <ae@op5.se>
Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
