From 74bcb63fae8595a181730f3c86ca9eeefb680053 Mon Sep 17 00:00:00 2001 From: Tim Clem Date: Mon, 7 Mar 2011 10:13:56 -0800 Subject: [PATCH 01/81] add objective-git to readme --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 1b3c68085..a8a93ac07 100644 --- a/README.md +++ b/README.md @@ -117,6 +117,7 @@ Here are the bindings to libgit2 that are currently available: * php-git (PHP bindings) * luagit2 (Lua bindings) * GitForDelphi (Delphi bindings) +* objective-git (Objective-C bindings) * Geef (Erlang bindings) If you start another language binding to libgit2, please let us know so From b760fbf539e1ce17ac2768da755834babf700b9a Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Mon, 7 Mar 2011 21:18:05 +0200 Subject: [PATCH 02/81] Update README.md with new bindings Signed-off-by: Vicent Marti --- README.md | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index a8a93ac07..5b27cfdcc 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,11 @@ libgit2 is a portable, pure C implementation of the Git core methods provided as re-entrant linkable library with a solid API, allowing you to write native speed custom Git applications in any language with bindings. +libgit2 is licensed under a **very permissive license** (GPLv2 with a special Linking Exception). +This basically means that you can link it (unmodified) with any kind of software without having to +release its source code. + +* Mailing list: * Website: * API documentation: * Usage guide: @@ -14,13 +19,17 @@ What It Can Do libgit2 is already very usable. -* raw <-> hex SHA conversions -* raw object reading (loose and packed) -* raw object writing (loose) -* revlist walker -* commit, tag and tree object parsing and write-back +* SHA conversions, formatting and shortening +* object reading (loose and packed) +* object writing (loose) +* commit, tag, tree and blob parsing and write-back * tree traversal -* basic index file (staging area) operations +* revision walking +* index file (staging area) manipulation +* custom ODB backends +* reference management (including packed references) +* ...and more + Building libgit2 - External dependencies ======================================== @@ -112,12 +121,14 @@ Language Bindings Here are the bindings to libgit2 that are currently available: * Rugged (Ruby bindings) -* pygit2 (Python bindings) -* libgit2sharp (.NET bindings) -* php-git (PHP bindings) -* luagit2 (Lua bindings) -* GitForDelphi (Delphi bindings) * objective-git (Objective-C bindings) +* pygit2 (Python bindings) +* libgit2sharp (.NET bindings) +* php-git (PHP bindings) +* luagit2 (Lua bindings) +* GitForDelphi (Delphi bindings) +* node-gitteh (Node.js bindings) +* libqgit2 (C++ QT bindings) * Geef (Erlang bindings) If you start another language binding to libgit2, please let us know so From 26022f0719f652ae8311abea6f6de92bd4a75a87 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Sat, 5 Mar 2011 23:54:49 +0200 Subject: [PATCH 03/81] Add `git_oid_shorten` (unique OID minimzer) Set of methods to find the minimal-length to uniquely identify every OID in a list. Useful for GUI applications, commit logs and so on. Includes stress test. Signed-off-by: Vicent Marti --- include/git2/oid.h | 54 ++++++++++++++ src/oid.c | 178 +++++++++++++++++++++++++++++++++++++++++++++ tests/t01-rawobj.c | 92 +++++++++++++++++++++++ 3 files changed, 324 insertions(+) diff --git a/include/git2/oid.h b/include/git2/oid.h index 5cac46f3b..4538c6147 100644 --- a/include/git2/oid.h +++ b/include/git2/oid.h @@ -132,6 +132,60 @@ GIT_EXTERN(void) git_oid_cpy(git_oid *out, const git_oid *src); */ GIT_EXTERN(int) git_oid_cmp(const git_oid *a, const git_oid *b); +/** + * OID Shortener object + */ +typedef struct git_oid_shorten git_oid_shorten; + +/** + * Create a new OID shortener. + * + * The OID shortener is used to process a list of OIDs + * in text form and return the shortest length that would + * uniquely identify all of them. + * + * E.g. look at the result of `git log --abbrev`. + * + * @param min_length The minimal length for all identifiers, + * which will be used even if shorter OIDs would still + * be unique. + * @return a `git_oid_shorten` instance, NULL if OOM + */ +git_oid_shorten *git_oid_shorten_new(size_t min_length); + +/** + * Add a new OID to set of shortened OIDs and calculate + * the minimal length to uniquely identify all the OIDs in + * the set. + * + * The OID is expected to be a 40-char hexadecimal string. + * The OID is owned by the user and will not be modified + * or freed. + * + * For performance reasons, there is a hard-limit of how many + * OIDs can be added to a single set (around ~22000, assuming + * a mostly randomized distribution), which should be enough + * for any kind of program, and keeps the algorithm fast and + * memory-efficient. + * + * Attempting to add more than those OIDs will result in a + * GIT_ENOMEM error + * + * @param os a `git_oid_shorten` instance + * @param text_oid an OID in text form + * @return the minimal length to uniquely identify all OIDs + * added so far to the set; or an error code (<0) if an + * error occurs. + */ +int git_oid_shorten_add(git_oid_shorten *os, const char *text_oid); + +/** + * Free an OID shortener instance + * + * @param os a `git_oid_shorten` instance + */ +void git_oid_shorten_free(git_oid_shorten *os); + /** @} */ GIT_END_DECL #endif diff --git a/src/oid.c b/src/oid.c index 698d0f927..81b7d6005 100644 --- a/src/oid.c +++ b/src/oid.c @@ -27,6 +27,7 @@ #include "git2/oid.h" #include "repository.h" #include +#include static signed char from_hex[] = { -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, /* 00 */ @@ -166,3 +167,180 @@ int git_oid_cmp(const git_oid *a, const git_oid *b) { return memcmp(a->id, b->id, sizeof(a->id)); } + + +typedef short node_index; + +typedef union { + const char *tail; + node_index children[16]; +} trie_node; + +struct git_oid_shorten { + trie_node *nodes; + size_t node_count, size; + int min_length, full; +}; + +static int resize_trie(git_oid_shorten *self, size_t new_size) +{ + self->nodes = realloc(self->nodes, new_size * sizeof(trie_node)); + if (self->nodes == NULL) + return GIT_ENOMEM; + + if (new_size > self->size) { + memset(&self->nodes[self->size], 0x0, (new_size - self->size) * sizeof(trie_node)); + } + + self->size = new_size; + return GIT_SUCCESS; +} + +static trie_node *push_leaf(git_oid_shorten *os, node_index idx, int push_at, const char *oid) +{ + trie_node *node, *leaf; + node_index idx_leaf; + + if (os->node_count >= os->size) { + if (resize_trie(os, os->size * 2) < GIT_SUCCESS) + return NULL; + } + + idx_leaf = (node_index)os->node_count++; + + if (os->node_count == SHRT_MAX) + os->full = 1; + + node = &os->nodes[idx]; + node->children[push_at] = -idx_leaf; + + leaf = &os->nodes[idx_leaf]; + leaf->tail = oid; + + return node; +} + +git_oid_shorten *git_oid_shorten_new(size_t min_length) +{ + git_oid_shorten *os; + + os = git__malloc(sizeof(git_oid_shorten)); + if (os == NULL) + return NULL; + + memset(os, 0x0, sizeof(git_oid_shorten)); + + if (resize_trie(os, 16) < GIT_SUCCESS) { + free(os); + return NULL; + } + + os->node_count = 1; + os->min_length = min_length; + + return os; +} + +void git_oid_shorten_free(git_oid_shorten *os) +{ + free(os->nodes); + free(os); +} + + +/* + * What wizardry is this? + * + * This is just a memory-optimized trie: basically a very fancy + * 16-ary tree, which is used to store the prefixes of the OID + * strings. + * + * Read more: http://en.wikipedia.org/wiki/Trie + * + * Magic that happens in this method: + * + * - Each node in the trie is an union, so it can work both as + * a normal node, or as a leaf. + * + * - Each normal node points to 16 children (one for each possible + * character in the oid). This is *not* stored in an array of + * pointers, because in a 64-bit arch this would be sucking + * 16*sizeof(void*) = 128 bytes of memory per node, which is fucking + * insane. What we do is store Node Indexes, and use these indexes + * to look up each node in the om->index array. These indexes are + * signed shorts, so this limits the amount of unique OIDs that + * fit in the structure to about 20000 (assuming a more or less uniform + * distribution). + * + * - All the nodes in om->index array are stored contiguously in + * memory, and each of them is 32 bytes, so we fit 2x nodes per + * cache line. Convenient for speed. + * + * - To differentiate the leafs from the normal nodes, we store all + * the indexes towards a leaf as a negative index (indexes to normal + * nodes are positives). When we find that one of the children for + * a node has a negative value, that means it's going to be a leaf. + * This reduces the amount of indexes we have by two, but also reduces + * the size of each node by 1-4 bytes (the amount we would need to + * add a `is_leaf` field): this is good because it allows the nodes + * to fit cleanly in cache lines. + * + * - Once we reach an empty children, instead of continuing to insert + * new nodes for each remaining character of the OID, we store a pointer + * to the tail in the leaf; if the leaf is reached again, we turn it + * into a normal node and use the tail to create a new leaf. + * + * This is a pretty good balance between performance and memory usage. + */ +int git_oid_shorten_add(git_oid_shorten *os, const char *text_oid) +{ + int i, is_leaf; + node_index idx; + + if (os->full) + return GIT_ENOMEM; + + idx = 0; + is_leaf = 0; + + for (i = 0; i < GIT_OID_HEXSZ; ++i) { + int c = from_hex[(int)text_oid[i]]; + trie_node *node; + + if (c == -1) + return GIT_ENOTOID; + + node = &os->nodes[idx]; + + if (is_leaf) { + const char *tail; + + tail = node->tail; + node->tail = NULL; + + node = push_leaf(os, idx, from_hex[(int)tail[0]], &tail[1]); + if (node == NULL) + return GIT_ENOMEM; + } + + if (node->children[c] == 0) { + if (push_leaf(os, idx, c, &text_oid[i + 1]) == NULL) + return GIT_ENOMEM; + break; + } + + idx = node->children[c]; + is_leaf = 0; + + if (idx < 0) { + node->children[c] = idx = -idx; + is_leaf = 1; + } + } + + if (++i > os->min_length) + os->min_length = i; + + return os->min_length; +} + diff --git a/tests/t01-rawobj.c b/tests/t01-rawobj.c index cc4641589..3dfa3c9fe 100644 --- a/tests/t01-rawobj.c +++ b/tests/t01-rawobj.c @@ -300,6 +300,96 @@ BEGIN_TEST(oid15, "convert raw oid to string (big)") must_be_true(str && str == big && *(str+GIT_OID_HEXSZ+3) == 'Z'); END_TEST + +BEGIN_TEST(oid16, "make sure the OID shortener doesn't choke on duplicate sha1s") + + git_oid_shorten *os; + int min_len; + + os = git_oid_shorten_new(0); + must_be_true(os != NULL); + + git_oid_shorten_add(os, "22596363b3de40b06f981fb85d82312e8c0ed511"); + git_oid_shorten_add(os, "ce08fe4884650f067bd5703b6a59a8b3b3c99a09"); + git_oid_shorten_add(os, "16a0123456789abcdef4b775213c23a8bd74f5e0"); + min_len = git_oid_shorten_add(os, "ce08fe4884650f067bd5703b6a59a8b3b3c99a09"); + + must_be_true(min_len == GIT_OID_HEXSZ + 1); + + git_oid_shorten_free(os); +END_TEST + +BEGIN_TEST(oid17, "stress test for the git_oid_shorten object") + +#define MAX_OIDS 1000 + + git_oid_shorten *os; + char *oids[MAX_OIDS]; + char number_buffer[16]; + git_oid oid; + size_t i, j; + + int min_len = 0, found_collision; + + os = git_oid_shorten_new(0); + must_be_true(os != NULL); + + /* + * Insert in the shortener 1000 unique SHA1 ids + */ + for (i = 0; i < MAX_OIDS; ++i) { + char *oid_text; + + sprintf(number_buffer, "%u", (unsigned int)i); + git_hash_buf(&oid, number_buffer, strlen(number_buffer)); + + oid_text = git__malloc(GIT_OID_HEXSZ + 1); + git_oid_fmt(oid_text, &oid); + oid_text[GIT_OID_HEXSZ] = 0; + + min_len = git_oid_shorten_add(os, oid_text); + must_be_true(min_len >= 0); + + oids[i] = oid_text; + } + + /* + * Compare the first `min_char - 1` characters of each + * SHA1 OID. If the minimizer worked, we should find at + * least one collision + */ + found_collision = 0; + for (i = 0; i < MAX_OIDS; ++i) { + for (j = 0; j < MAX_OIDS; ++j) { + if (i != j && memcmp(oids[i], oids[j], min_len - 1) == 0) + found_collision = 1; + } + } + must_be_true(found_collision == 1); + + /* + * Compare the first `min_char` characters of each + * SHA1 OID. If the minimizer worked, every single preffix + * should be unique. + */ + found_collision = 0; + for (i = 0; i < MAX_OIDS; ++i) { + for (j = 0; j < MAX_OIDS; ++j) { + if (i != j && memcmp(oids[i], oids[j], min_len) == 0) + found_collision = 1; + } + } + must_be_true(found_collision == 0); + + /* cleanup */ + for (i = 0; i < MAX_OIDS; ++i) + free(oids[i]); + + git_oid_shorten_free(os); + +#undef MAX_OIDS +END_TEST + static char *hello_id = "22596363b3de40b06f981fb85d82312e8c0ed511"; static char *hello_text = "hello world\n"; @@ -518,6 +608,8 @@ BEGIN_SUITE(rawobjects) ADD_TEST(oid13); ADD_TEST(oid14); ADD_TEST(oid15); + ADD_TEST(oid16); + ADD_TEST(oid17); ADD_TEST(hash0); ADD_TEST(hash1); From 71db842fac3ba8582255bc5b61361ddef08ef105 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 8 Mar 2011 14:57:03 +0200 Subject: [PATCH 04/81] Rewrite the Revision Walker The new revision walker uses an internal Commit object storage system, custom memory allocator and much improved topological and time sorting algorithms. It's about 20x times faster than the previous implementation when browsing big repositories. The following external API calls have changed: `git_revwalk_next` returns an OID instead of a full commit object. The initial call to `git_revwalk_next` is no longer blocking when iterating through a repo with a time-sorting mode. Iterating with Topological or inverted modes still makes the initial call blocking to preprocess the commit list, but this block should be mostly unnoticeable on most repositories (topological preprocessing times at 0.3s on the git.git repo). `git_revwalk_push` and `git_revwalk_hide` now take an OID instead of a full commit object. --- include/git2/revwalk.h | 8 +- src/pqueue.c | 153 +++++++++ src/pqueue.h | 92 +++++ src/repository.c | 2 +- src/revwalk.c | 744 +++++++++++++++++++++++------------------ src/revwalk.h | 56 ---- src/util.h | 2 + tests/t05-revwalk.c | 109 ++---- 8 files changed, 699 insertions(+), 467 deletions(-) create mode 100644 src/pqueue.c create mode 100644 src/pqueue.h diff --git a/include/git2/revwalk.h b/include/git2/revwalk.h index 841110499..fdbbe236c 100644 --- a/include/git2/revwalk.h +++ b/include/git2/revwalk.h @@ -27,6 +27,7 @@ #include "common.h" #include "types.h" +#include "object.h" /** * @file git2/revwalk.h @@ -88,14 +89,15 @@ GIT_EXTERN(void) git_revwalk_reset(git_revwalk *walker); * @param walker the walker being used for the traversal. * @param commit the commit to start from. */ -GIT_EXTERN(int) git_revwalk_push(git_revwalk *walk, git_commit *commit); +GIT_EXTERN(int) git_revwalk_push(git_revwalk *walk, const git_oid *oid); + /** * Mark a commit (and its ancestors) uninteresting for the output. * @param walker the walker being used for the traversal. * @param commit the commit that will be ignored during the traversal */ -GIT_EXTERN(int) git_revwalk_hide(git_revwalk *walk, git_commit *commit); +GIT_EXTERN(int) git_revwalk_hide(git_revwalk *walk, const git_oid *oid); /** * Get the next commit from the revision traversal. @@ -105,7 +107,7 @@ GIT_EXTERN(int) git_revwalk_hide(git_revwalk *walk, git_commit *commit); * @return GIT_SUCCESS if the next commit was found; * GIT_EREVWALKOVER if there are no commits left to iterate */ -GIT_EXTERN(int) git_revwalk_next(git_commit **commit, git_revwalk *walk); +GIT_EXTERN(int) git_revwalk_next(git_oid *oid, git_revwalk *walk); /** * Change the sorting mode when iterating through the diff --git a/src/pqueue.c b/src/pqueue.c new file mode 100644 index 000000000..98152cb85 --- /dev/null +++ b/src/pqueue.c @@ -0,0 +1,153 @@ +/* + * BORING COPYRIGHT NOTICE: + * + * This file is a heavily modified version of the priority queue found + * in the Apache project and the libpqueue library. + * + * https://github.com/vy/libpqueue + * + * These are the original authors: + * + * Copyright 2010 Volkan Yazıcı + * Copyright 2006-2010 The Apache Software Foundation + * + * This file is licensed under the Apache 2.0 license, which + * supposedly makes it compatible with the GPLv2 that libgit2 uses. + * + * Check the Apache license at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * So much licensing trouble for a binary heap. Oh well. + */ + +#include "common.h" +#include "pqueue.h" + +#define left(i) ((i) << 1) +#define right(i) (((i) << 1) + 1) +#define parent(i) ((i) >> 1) + +int git_pqueue_init(git_pqueue *q, size_t n, git_pqueue_cmp cmppri) +{ + assert(q); + + /* Need to allocate n+1 elements since element 0 isn't used. */ + if ((q->d = malloc((n + 1) * sizeof(void *))) == NULL) + return GIT_ENOMEM; + + q->size = 1; + q->avail = q->step = (n + 1); /* see comment above about n+1 */ + q->cmppri = cmppri; + + return GIT_SUCCESS; +} + + +void git_pqueue_free(git_pqueue *q) +{ + free(q->d); + q->d = NULL; +} + + +size_t git_pqueue_size(git_pqueue *q) +{ + /* queue element 0 exists but doesn't count since it isn't used. */ + return (q->size - 1); +} + + +static void bubble_up(git_pqueue *q, size_t i) +{ + size_t parent_node; + void *moving_node = q->d[i]; + + for (parent_node = parent(i); + ((i > 1) && q->cmppri(q->d[parent_node], moving_node)); + i = parent_node, parent_node = parent(i)) { + q->d[i] = q->d[parent_node]; + } + + q->d[i] = moving_node; +} + + +static size_t maxchild(git_pqueue *q, size_t i) +{ + size_t child_node = left(i); + + if (child_node >= q->size) + return 0; + + if ((child_node + 1) < q->size && + q->cmppri(q->d[child_node], q->d[child_node + 1])) + child_node++; /* use right child instead of left */ + + return child_node; +} + + +static void percolate_down(git_pqueue *q, size_t i) +{ + size_t child_node; + void *moving_node = q->d[i]; + + while ((child_node = maxchild(q, i)) != 0 && + q->cmppri(moving_node, q->d[child_node])) { + q->d[i] = q->d[child_node]; + i = child_node; + } + + q->d[i] = moving_node; +} + + +int git_pqueue_insert(git_pqueue *q, void *d) +{ + void *tmp; + size_t i; + size_t newsize; + + if (!q) return 1; + + /* allocate more memory if necessary */ + if (q->size >= q->avail) { + newsize = q->size + q->step; + if ((tmp = realloc(q->d, sizeof(void *) * newsize)) == NULL) + return GIT_ENOMEM; + + q->d = tmp; + q->avail = newsize; + } + + /* insert item */ + i = q->size++; + q->d[i] = d; + bubble_up(q, i); + + return GIT_SUCCESS; +} + + +void *git_pqueue_pop(git_pqueue *q) +{ + void *head; + + if (!q || q->size == 1) + return NULL; + + head = q->d[1]; + q->d[1] = q->d[--q->size]; + percolate_down(q, 1); + + return head; +} + + +void *git_pqueue_peek(git_pqueue *q) +{ + if (!q || q->size == 1) + return NULL; + return q->d[1]; +} diff --git a/src/pqueue.h b/src/pqueue.h new file mode 100644 index 000000000..6db74661d --- /dev/null +++ b/src/pqueue.h @@ -0,0 +1,92 @@ +/* + * BORING COPYRIGHT NOTICE: + * + * This file is a heavily modified version of the priority queue found + * in the Apache project and the libpqueue library. + * + * https://github.com/vy/libpqueue + * + * These are the original authors: + * + * Copyright 2010 Volkan Yazıcı + * Copyright 2006-2010 The Apache Software Foundation + * + * This file is licensed under the Apache 2.0 license, which + * supposedly makes it compatible with the GPLv2 that libgit2 uses. + * + * Check the Apache license at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * So much licensing trouble for a binary heap. Oh well. + */ + +#ifndef INCLUDE_pqueue_h__ +#define INCLUDE_pqueue_h__ + +/** callback functions to get/set/compare the priority of an element */ +typedef int (*git_pqueue_cmp)(void *a, void *b); + +/** the priority queue handle */ +typedef struct { + size_t size, avail, step; + git_pqueue_cmp cmppri; + void **d; +} git_pqueue; + + +/** + * initialize the queue + * + * @param n the initial estimate of the number of queue items for which memory + * should be preallocated + * @param cmppri the callback function to compare two nodes of the queue + * + * @Return the handle or NULL for insufficent memory + */ +int git_pqueue_init(git_pqueue *q, size_t n, git_pqueue_cmp cmppri); + + +/** + * free all memory used by the queue + * @param q the queue + */ +void git_pqueue_free(git_pqueue *q); + + +/** + * return the size of the queue. + * @param q the queue + */ +size_t git_pqueue_size(git_pqueue *q); + + +/** + * insert an item into the queue. + * @param q the queue + * @param d the item + * @return 0 on success + */ +int git_pqueue_insert(git_pqueue *q, void *d); + + +/** + * pop the highest-ranking item from the queue. + * @param p the queue + * @param d where to copy the entry to + * @return NULL on error, otherwise the entry + */ +void *git_pqueue_pop(git_pqueue *q); + + +/** + * access highest-ranking item without removing it. + * @param q the queue + * @param d the entry + * @return NULL on error, otherwise the entry + */ +void *git_pqueue_peek(git_pqueue *q); + +#endif /* PQUEUE_H */ +/** @} */ + diff --git a/src/repository.c b/src/repository.c index f2cb985af..c24560665 100644 --- a/src/repository.c +++ b/src/repository.c @@ -53,7 +53,7 @@ typedef struct { * Callbacks for the ODB cache, implemented * as a hash table */ -uint32_t object_table_hash(const void *key, int hash_id) +static uint32_t object_table_hash(const void *key, int hash_id) { uint32_t r; git_oid *id; diff --git a/src/revwalk.c b/src/revwalk.c index a1cd0ebb7..edafbe73d 100644 --- a/src/revwalk.c +++ b/src/revwalk.c @@ -27,22 +27,132 @@ #include "commit.h" #include "revwalk.h" #include "hashtable.h" +#include "pqueue.h" -uint32_t git_revwalk__commit_hash(const void *key, int hash_id) +typedef struct commit_object { + git_oid oid; + uint32_t time; + unsigned int seen:1, + uninteresting:1, + topo_delay:1, + parsed:1; + + unsigned short in_degree; + unsigned short out_degree; + + struct commit_object **parents; +} commit_object; + +typedef struct commit_list { + commit_object *item; + struct commit_list *next; +} commit_list; + +struct git_revwalk { + git_repository *repo; + + git_hashtable *commits; + git_vector pending; + + commit_list *iterator_topo; + commit_list *iterator_rand; + commit_list *iterator_reverse; + git_pqueue iterator_time; + + int (*get_next)(commit_object **, git_revwalk *); + int (*enqueue)(git_revwalk *, commit_object *); + + git_vector memory_alloc; + size_t chunk_size; + + unsigned walking:1; + unsigned int sorting; +}; + +commit_list *commit_list_insert(commit_object *item, commit_list **list_p) +{ + commit_list *new_list = git__malloc(sizeof(commit_list)); + new_list->item = item; + new_list->next = *list_p; + *list_p = new_list; + return new_list; +} + +void commit_list_free(commit_list *list) +{ + while (list) { + commit_list *temp = list; + list = temp->next; + free(temp); + } +} + +commit_object *commit_list_pop(commit_list **stack) +{ + commit_list *top = *stack; + commit_object *item = top ? top->item : NULL; + + if (top) { + *stack = top->next; + free(top); + } + return item; +} + +static int commit_time_cmp(void *a, void *b) +{ + commit_object *commit_a = (commit_object *)a; + commit_object *commit_b = (commit_object *)b; + + return (commit_a->time < commit_b->time); +} + +static uint32_t object_table_hash(const void *key, int hash_id) { uint32_t r; - git_commit *commit; + git_oid *id; - commit = (git_commit *)key; - memcpy(&r, commit->object.id.id + (hash_id * sizeof(uint32_t)), sizeof(r)); + id = (git_oid *)key; + memcpy(&r, id->id + (hash_id * sizeof(uint32_t)), sizeof(r)); return r; } -int git_revwalk__commit_keycmp(const void *key_a, const void *key_b) +#define COMMITS_PER_CHUNK 128 +#define CHUNK_STEP 64 +#define PARENTS_PER_COMMIT ((CHUNK_STEP - sizeof(commit_object)) / sizeof(commit_object *)) + +static int alloc_chunk(git_revwalk *walk) { - git_commit *a = (git_commit *)key_a; - git_commit *b = (git_commit *)key_b; - return git_oid_cmp(&a->object.id, &b->object.id); + void *chunk; + + chunk = git__calloc(COMMITS_PER_CHUNK, CHUNK_STEP); + if (chunk == NULL) + return GIT_ENOMEM; + + walk->chunk_size = 0; + return git_vector_insert(&walk->memory_alloc, chunk); +} + +static commit_object *alloc_commit(git_revwalk *walk) +{ + unsigned char *chunk; + + if (walk->chunk_size == COMMITS_PER_CHUNK) + alloc_chunk(walk); + + chunk = git_vector_get(&walk->memory_alloc, walk->memory_alloc.length - 1); + chunk += (walk->chunk_size * CHUNK_STEP); + walk->chunk_size++; + + return (commit_object *)chunk; +} + +static commit_object **alloc_parents(commit_object *commit, size_t n_parents) +{ + if (n_parents <= PARENTS_PER_COMMIT) + return (commit_object **)((unsigned char *)commit + sizeof(commit_object)); + + return git__malloc(n_parents * sizeof(commit_object *)); } int git_revwalk_new(git_revwalk **revwalk_out, git_repository *repo) @@ -56,14 +166,18 @@ int git_revwalk_new(git_revwalk **revwalk_out, git_repository *repo) memset(walk, 0x0, sizeof(git_revwalk)); walk->commits = git_hashtable_alloc(64, - git_revwalk__commit_hash, - git_revwalk__commit_keycmp); + object_table_hash, + (git_hash_keyeq_ptr)git_oid_cmp); if (walk->commits == NULL) { free(walk); return GIT_ENOMEM; } + git_vector_init(&walk->pending, 8, NULL); + git_vector_init(&walk->memory_alloc, 8, NULL); + alloc_chunk(walk); + walk->repo = repo; *revwalk_out = walk; @@ -72,11 +186,20 @@ int git_revwalk_new(git_revwalk **revwalk_out, git_repository *repo) void git_revwalk_free(git_revwalk *walk) { + unsigned int i; + if (walk == NULL) return; git_revwalk_reset(walk); git_hashtable_free(walk->commits); + git_vector_free(&walk->pending); + + for (i = 0; i < walk->memory_alloc.length; ++i) { + free(git_vector_get(&walk->memory_alloc, i)); + } + + git_vector_free(&walk->memory_alloc); free(walk); } @@ -98,364 +221,349 @@ int git_revwalk_sorting(git_revwalk *walk, unsigned int sort_mode) return GIT_SUCCESS; } -static git_revwalk_commit *commit_to_walkcommit(git_revwalk *walk, git_commit *commit_object) +static commit_object *commit_lookup(git_revwalk *walk, const git_oid *oid) { - git_revwalk_commit *commit; + commit_object *commit; - commit = (git_revwalk_commit *)git_hashtable_lookup(walk->commits, commit_object); - - if (commit != NULL) + if ((commit = git_hashtable_lookup(walk->commits, oid)) != NULL) return commit; - commit = git__malloc(sizeof(git_revwalk_commit)); + commit = alloc_commit(walk); if (commit == NULL) return NULL; - memset(commit, 0x0, sizeof(git_revwalk_commit)); + git_oid_cpy(&commit->oid, oid); - commit->commit_object = commit_object; - GIT_OBJECT_INCREF(walk->repo, commit_object); - - git_hashtable_insert(walk->commits, commit_object, commit); + if (git_hashtable_insert(walk->commits, &commit->oid, commit) < GIT_SUCCESS) { + free(commit); + return NULL; + } return commit; } -static git_revwalk_commit *insert_commit(git_revwalk *walk, git_commit *commit_object) +static int commit_quick_parse(git_revwalk *walk, commit_object *commit, git_rawobj *raw) { - git_revwalk_commit *commit; - unsigned int i; + const int parent_len = STRLEN("parent ") + GIT_OID_HEXSZ + 1; - assert(walk && commit_object); + unsigned char *buffer = raw->data; + unsigned char *buffer_end = buffer + raw->len; + unsigned char *parents_start; - if (commit_object->object.repo != walk->repo || walk->walking) - return NULL; + int i, parents = 0; - commit = commit_to_walkcommit(walk, commit_object); - if (commit == NULL) - return NULL; + buffer += STRLEN("tree ") + GIT_OID_HEXSZ + 1; - if (commit->seen) - return commit; - - commit->seen = 1; - - for (i = 0; i < commit->commit_object->parents.length; ++i) { - git_commit *parent_object; - git_revwalk_commit *parent; - - parent_object = git_vector_get(&commit->commit_object->parents, i); - - if ((parent = commit_to_walkcommit(walk, parent_object)) == NULL) - return NULL; - - parent = insert_commit(walk, parent_object); - if (parent == NULL) - return NULL; - - parent->in_degree++; - - git_revwalk_list_push_back(&commit->parents, parent); + parents_start = buffer; + while (buffer + parent_len < buffer_end && memcmp(buffer, "parent ", STRLEN("parent ")) == 0) { + parents++; + buffer += parent_len; } - if (git_revwalk_list_push_back(&walk->iterator, commit)) - return NULL; - - return commit; -} - -int git_revwalk_push(git_revwalk *walk, git_commit *commit) -{ - assert(walk && commit); - return insert_commit(walk, commit) ? GIT_SUCCESS : GIT_ENOMEM; -} - -static void mark_uninteresting(git_revwalk_commit *commit) -{ - git_revwalk_listnode *parent; - - assert(commit); - - commit->uninteresting = 1; - parent = commit->parents.head; - - while (parent) { - mark_uninteresting(parent->walk_commit); - parent = parent->next; - } -} - -int git_revwalk_hide(git_revwalk *walk, git_commit *commit) -{ - git_revwalk_commit *hide; - - assert(walk && commit); - - hide = insert_commit(walk, commit); - if (hide == NULL) + commit->parents = alloc_parents(commit, parents); + if (commit->parents == NULL) return GIT_ENOMEM; - mark_uninteresting(hide); + buffer = parents_start; + for (i = 0; i < parents; ++i) { + git_oid oid; + + if (git_oid_mkstr(&oid, (char *)buffer + STRLEN("parent ")) < GIT_SUCCESS) + return GIT_EOBJCORRUPTED; + + commit->parents[i] = commit_lookup(walk, &oid); + if (commit->parents[i] == NULL) + return GIT_ENOMEM; + + buffer += parent_len; + } + + commit->out_degree = (unsigned short)parents; + + if ((buffer = memchr(buffer, '\n', buffer_end - buffer)) == NULL) + return GIT_EOBJCORRUPTED; + + buffer = memchr(buffer, '>', buffer_end - buffer); + if (buffer == NULL) + return GIT_EOBJCORRUPTED; + + commit->time = strtol((char *)buffer + 2, NULL, 10); + if (commit->time == 0) + return GIT_EOBJCORRUPTED; + + commit->parsed = 1; return GIT_SUCCESS; } - -static void prepare_walk(git_revwalk *walk) +static int commit_parse(git_revwalk *walk, commit_object *commit) { - if (walk->sorting & GIT_SORT_TIME) - git_revwalk_list_timesort(&walk->iterator); + git_rawobj data; + int error; - if (walk->sorting & GIT_SORT_TOPOLOGICAL) - git_revwalk_list_toposort(&walk->iterator); + if (commit->parsed) + return GIT_SUCCESS; - if (walk->sorting & GIT_SORT_REVERSE) - walk->next = &git_revwalk_list_pop_back; - else - walk->next = &git_revwalk_list_pop_front; + if ((error = git_odb_read(&data, walk->repo->db, &commit->oid)) < GIT_SUCCESS) + return error; - walk->walking = 1; + if (data.type != GIT_OBJ_COMMIT) { + git_rawobj_close(&data); + return GIT_EOBJTYPE; + } + + error = commit_quick_parse(walk, commit, &data); + git_rawobj_close(&data); + return error; } -int git_revwalk_next(git_commit **commit, git_revwalk *walk) +static void mark_uninteresting(commit_object *commit) { - git_revwalk_commit *next; + unsigned short i; + assert(commit); - assert(walk && commit); + commit->uninteresting = 1; - if (!walk->walking) - prepare_walk(walk); + for (i = 0; i < commit->out_degree; ++i) + if (!commit->parents[i]->uninteresting) + mark_uninteresting(commit->parents[i]); +} - *commit = NULL; +static int process_commit(git_revwalk *walk, commit_object *commit) +{ + int error; + + if (commit->seen) + return GIT_SUCCESS; + + commit->seen = 1; + + if ((error = commit_parse(walk, commit)) < GIT_SUCCESS) + return error; + + if (commit->uninteresting) + mark_uninteresting(commit); + + return walk->enqueue(walk, commit); +} + +static int process_commit_parents(git_revwalk *walk, commit_object *commit) +{ + unsigned short i; + int error = GIT_SUCCESS; + + for (i = 0; i < commit->out_degree && error == GIT_SUCCESS; ++i) { + error = process_commit(walk, commit->parents[i]); + } + + return error; +} + +static int push_commit(git_revwalk *walk, const git_oid *oid, int uninteresting) +{ + commit_object *commit; + + commit = commit_lookup(walk, oid); + if (commit == NULL) + return GIT_ENOTFOUND; + + if (uninteresting) + mark_uninteresting(commit); + + return git_vector_insert(&walk->pending, commit); +} + +int git_revwalk_push(git_revwalk *walk, const git_oid *oid) +{ + assert(walk && oid); + return push_commit(walk, oid, 0); +} + +int git_revwalk_hide(git_revwalk *walk, const git_oid *oid) +{ + assert(walk && oid); + return push_commit(walk, oid, 1); +} + +static int revwalk_enqueue_timesort(git_revwalk *walk, commit_object *commit) +{ + return git_pqueue_insert(&walk->iterator_time, commit); +} + +static int revwalk_enqueue_unsorted(git_revwalk *walk, commit_object *commit) +{ + return commit_list_insert(commit, &walk->iterator_rand) ? GIT_SUCCESS : GIT_ENOMEM; +} + +static int revwalk_next_timesort(commit_object **object_out, git_revwalk *walk) +{ + int error; + commit_object *next; + + while ((next = git_pqueue_pop(&walk->iterator_time)) != NULL) { + if ((error = process_commit_parents(walk, next)) < GIT_SUCCESS) + return error; - while ((next = walk->next(&walk->iterator)) != NULL) { if (!next->uninteresting) { - *commit = next->commit_object; - GIT_OBJECT_INCREF(walk->repo, *commit); + *object_out = next; return GIT_SUCCESS; } } - /* No commits left to iterate */ - git_revwalk_reset(walk); return GIT_EREVWALKOVER; } +static int revwalk_next_unsorted(commit_object **object_out, git_revwalk *walk) +{ + int error; + commit_object *next; + + while ((next = commit_list_pop(&walk->iterator_rand)) != NULL) { + if ((error = process_commit_parents(walk, next)) < GIT_SUCCESS) + return error; + + if (!next->uninteresting) { + *object_out = next; + return GIT_SUCCESS; + } + } + + return GIT_EREVWALKOVER; +} + +static int revwalk_next_toposort(commit_object **object_out, git_revwalk *walk) +{ + commit_object *next; + unsigned short i; + + for (;;) { + next = commit_list_pop(&walk->iterator_topo); + if (next == NULL) + return GIT_EREVWALKOVER; + + if (next->in_degree > 0) { + next->topo_delay = 1; + continue; + } + + for (i = 0; i < next->out_degree; ++i) { + commit_object *parent = next->parents[i]; + + if (--parent->in_degree == 0 && parent->topo_delay) { + parent->topo_delay = 0; + commit_list_insert(parent, &walk->iterator_topo); + } + } + + *object_out = next; + return GIT_SUCCESS; + } +} + +static int revwalk_next_reverse(commit_object **object_out, git_revwalk *walk) +{ + *object_out = commit_list_pop(&walk->iterator_reverse); + return *object_out ? GIT_SUCCESS : GIT_EREVWALKOVER; +} + + +static int prepare_walk(git_revwalk *walk) +{ + unsigned int i; + int error; + + if (walk->sorting & GIT_SORT_TIME) { + if ((error = git_pqueue_init(&walk->iterator_time, 32, commit_time_cmp)) < GIT_SUCCESS) + return error; + + walk->get_next = &revwalk_next_timesort; + walk->enqueue = &revwalk_enqueue_timesort; + } else { + walk->get_next = &revwalk_next_unsorted; + walk->enqueue = &revwalk_enqueue_unsorted; + } + + for (i = 0; i < walk->pending.length; ++i) { + commit_object *commit = walk->pending.contents[i]; + if ((error = process_commit(walk, commit)) < GIT_SUCCESS) { + return error; + } + } + + if (walk->sorting & GIT_SORT_TOPOLOGICAL) { + commit_object *next; + unsigned short i; + int error; + + while ((error = walk->get_next(&next, walk)) == GIT_SUCCESS) { + for (i = 0; i < next->out_degree; ++i) { + commit_object *parent = next->parents[i]; + parent->in_degree++; + } + + commit_list_insert(next, &walk->iterator_topo); + } + + if (error != GIT_EREVWALKOVER) + return error; + + walk->get_next = &revwalk_next_toposort; + } + + if (walk->sorting & GIT_SORT_REVERSE) { + commit_object *next; + int error; + + while ((error = walk->get_next(&next, walk)) == GIT_SUCCESS) + commit_list_insert(next, &walk->iterator_reverse); + + if (error != GIT_EREVWALKOVER) + return error; + + walk->get_next = &revwalk_next_reverse; + } + + walk->walking = 1; + return GIT_SUCCESS; +} + + +int git_revwalk_next(git_oid *oid, git_revwalk *walk) +{ + int error; + commit_object *next; + + assert(walk && oid); + + if (!walk->walking) { + if ((error = prepare_walk(walk)) < GIT_SUCCESS) + return error; + } + + error = walk->get_next(&next, walk); + if (error < GIT_SUCCESS) + return error; + + git_oid_cpy(oid, &next->oid); + return GIT_SUCCESS; +} + void git_revwalk_reset(git_revwalk *walk) { const void *_unused; - git_revwalk_commit *commit; + commit_object *commit; assert(walk); - GIT_HASHTABLE_FOREACH(walk->commits, _unused, commit, { - GIT_OBJECT_DECREF(walk->repo, commit->commit_object); - git_revwalk_list_clear(&commit->parents); - free(commit); - }); + GIT_HASHTABLE_FOREACH(walk->commits, _unused, commit, + commit->seen = 0; + commit->in_degree = 0; + commit->topo_delay = 0; + ); - git_hashtable_clear(walk->commits); - git_revwalk_list_clear(&walk->iterator); + git_pqueue_free(&walk->iterator_time); + commit_list_free(walk->iterator_topo); + commit_list_free(walk->iterator_rand); + commit_list_free(walk->iterator_reverse); walk->walking = 0; } - - - - - -int git_revwalk_list_push_back(git_revwalk_list *list, git_revwalk_commit *commit) -{ - git_revwalk_listnode *node = NULL; - - node = git__malloc(sizeof(git_revwalk_listnode)); - - if (node == NULL) - return GIT_ENOMEM; - - node->walk_commit = commit; - node->next = NULL; - node->prev = list->tail; - - if (list->tail == NULL) { - list->head = list->tail = node; - } else { - list->tail->next = node; - list->tail = node; - } - - list->size++; - return 0; -} - -int git_revwalk_list_push_front(git_revwalk_list *list, git_revwalk_commit *commit) -{ - git_revwalk_listnode *node = NULL; - - node = git__malloc(sizeof(git_revwalk_listnode)); - - if (node == NULL) - return GIT_ENOMEM; - - node->walk_commit = commit; - node->next = list->head; - node->prev = NULL; - - if (list->head == NULL) { - list->head = list->tail = node; - } else { - list->head->prev = node; - list->head = node; - } - - list->size++; - return 0; -} - - -git_revwalk_commit *git_revwalk_list_pop_back(git_revwalk_list *list) -{ - git_revwalk_listnode *node; - git_revwalk_commit *commit; - - if (list->tail == NULL) - return NULL; - - node = list->tail; - list->tail = list->tail->prev; - if (list->tail == NULL) - list->head = NULL; - else - list->tail->next = NULL; - - commit = node->walk_commit; - free(node); - - list->size--; - - return commit; -} - -git_revwalk_commit *git_revwalk_list_pop_front(git_revwalk_list *list) -{ - git_revwalk_listnode *node; - git_revwalk_commit *commit; - - if (list->head == NULL) - return NULL; - - node = list->head; - list->head = list->head->next; - if (list->head == NULL) - list->tail = NULL; - else - list->head->prev = NULL; - - commit = node->walk_commit; - free(node); - - list->size--; - - return commit; -} - -void git_revwalk_list_clear(git_revwalk_list *list) -{ - git_revwalk_listnode *node, *next_node; - - node = list->head; - while (node) { - next_node = node->next; - free(node); - node = next_node; - } - - list->head = list->tail = NULL; - list->size = 0; -} - -void git_revwalk_list_timesort(git_revwalk_list *list) -{ - git_revwalk_listnode *p, *q, *e; - int in_size, p_size, q_size, merge_count, i; - - if (list->head == NULL) - return; - - in_size = 1; - - do { - p = list->head; - list->tail = NULL; - merge_count = 0; - - while (p != NULL) { - merge_count++; - q = p; - p_size = 0; - q_size = in_size; - - for (i = 0; i < in_size && q; ++i, q = q->next) - p_size++; - - while (p_size > 0 || (q_size > 0 && q)) { - - if (p_size == 0) - e = q, q = q->next, q_size--; - - else if (q_size == 0 || q == NULL || - p->walk_commit->commit_object->committer->when.time >= - q->walk_commit->commit_object->committer->when.time) - e = p, p = p->next, p_size--; - - else - e = q, q = q->next, q_size--; - - if (list->tail != NULL) - list->tail->next = e; - else - list->head = e; - - e->prev = list->tail; - list->tail = e; - } - - p = q; - } - - list->tail->next = NULL; - in_size *= 2; - - } while (merge_count > 1); -} - -void git_revwalk_list_toposort(git_revwalk_list *list) -{ - git_revwalk_commit *commit; - git_revwalk_list topo; - memset(&topo, 0x0, sizeof(git_revwalk_list)); - - while ((commit = git_revwalk_list_pop_back(list)) != NULL) { - git_revwalk_listnode *p; - - if (commit->in_degree > 0) { - commit->topo_delay = 1; - continue; - } - - for (p = commit->parents.head; p != NULL; p = p->next) { - p->walk_commit->in_degree--; - - if (p->walk_commit->in_degree == 0 && p->walk_commit->topo_delay) { - p->walk_commit->topo_delay = 0; - git_revwalk_list_push_back(list, p->walk_commit); - } - } - - git_revwalk_list_push_back(&topo, commit); - } - - list->head = topo.head; - list->tail = topo.tail; - list->size = topo.size; -} - diff --git a/src/revwalk.h b/src/revwalk.h index 7b69ccd63..2970d773c 100644 --- a/src/revwalk.h +++ b/src/revwalk.h @@ -8,60 +8,4 @@ #include "repository.h" #include "hashtable.h" -struct git_revwalk_commit; - -typedef struct git_revwalk_listnode { - struct git_revwalk_commit *walk_commit; - struct git_revwalk_listnode *next; - struct git_revwalk_listnode *prev; -} git_revwalk_listnode; - -typedef struct git_revwalk_list { - struct git_revwalk_listnode *head; - struct git_revwalk_listnode *tail; - size_t size; -} git_revwalk_list; - - -struct git_revwalk_commit { - - git_commit *commit_object; - git_revwalk_list parents; - - unsigned short in_degree; - unsigned seen:1, - uninteresting:1, - topo_delay:1, - flags:25; -}; - -typedef struct git_revwalk_commit git_revwalk_commit; - -struct git_revwalk { - git_repository *repo; - - git_hashtable *commits; - git_revwalk_list iterator; - - git_revwalk_commit *(*next)(git_revwalk_list *); - - unsigned walking:1; - unsigned int sorting; -}; - - -void git_revwalk__prepare_walk(git_revwalk *walk); -int git_revwalk__enroot(git_revwalk *walk, git_commit *commit); - -int git_revwalk_list_push_back(git_revwalk_list *list, git_revwalk_commit *commit); -int git_revwalk_list_push_front(git_revwalk_list *list, git_revwalk_commit *obj); - -git_revwalk_commit *git_revwalk_list_pop_back(git_revwalk_list *list); -git_revwalk_commit *git_revwalk_list_pop_front(git_revwalk_list *list); - -void git_revwalk_list_clear(git_revwalk_list *list); - -void git_revwalk_list_timesort(git_revwalk_list *list); -void git_revwalk_list_toposort(git_revwalk_list *list); - #endif /* INCLUDE_revwalk_h__ */ diff --git a/src/util.h b/src/util.h index d5320e15b..fab93c1c2 100644 --- a/src/util.h +++ b/src/util.h @@ -93,6 +93,8 @@ GIT_INLINE(int) git__is_sizet(git_off_t p) extern char *git__strtok(char *output, char *src, char *delimit); extern char *git__strtok_keep(char *output, char *src, char *delimit); +#define STRLEN(str) (sizeof(str) - 1) + /* * Realloc the buffer pointed at by variable 'x' so that it can hold * at least 'nr' entries; the number of entries currently allocated diff --git a/tests/t05-revwalk.c b/tests/t05-revwalk.c index fd009fac1..bdec09e83 100644 --- a/tests/t05-revwalk.c +++ b/tests/t05-revwalk.c @@ -70,12 +70,12 @@ static const int commit_sorting_time_reverse[][6] = { static const int result_bytes = 24; -static int get_commit_index(git_commit *commit) +static int get_commit_index(git_oid *raw_oid) { int i; char oid[40]; - git_oid_fmt(oid, &commit->object.id); + git_oid_fmt(oid, raw_oid); for (i = 0; i < commit_count; ++i) if (memcmp(oid, commit_ids[i], 40) == 0) @@ -84,23 +84,31 @@ static int get_commit_index(git_commit *commit) return -1; } -static int test_walk(git_revwalk *walk, git_commit *start_from, +static int test_walk(git_revwalk *walk, int flags, const int possible_results[][6], int results_count) { - git_commit *commit = NULL; + git_oid oid; int i; int result_array[commit_count]; + git_revwalk_reset(walk); git_revwalk_sorting(walk, flags); - git_revwalk_push(walk, start_from); for (i = 0; i < commit_count; ++i) result_array[i] = -1; i = 0; - while (git_revwalk_next(&commit, walk) == GIT_SUCCESS) - result_array[i++] = get_commit_index(commit); + + while (git_revwalk_next(&oid, walk) == GIT_SUCCESS) { + result_array[i++] = get_commit_index(&oid); + /*{ + char str[41]; + git_oid_fmt(str, &oid); + str[40] = 0; + printf(" %d) %s\n", i, str); + }*/ + } for (i = 0; i < results_count; ++i) if (memcmp(possible_results[i], @@ -114,103 +122,26 @@ BEGIN_TEST(walk0, "do a simple walk on a repo with different sorting modes") git_oid id; git_repository *repo; git_revwalk *walk; - git_commit *head = NULL; must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); must_pass(git_revwalk_new(&walk, repo)); git_oid_mkstr(&id, commit_head); + git_revwalk_push(walk, &id); - must_pass(git_commit_lookup(&head, repo, &id)); + must_pass(test_walk(walk, GIT_SORT_TIME, commit_sorting_time, 1)); - must_pass(test_walk(walk, head, - GIT_SORT_TIME, - commit_sorting_time, 1)); + must_pass(test_walk(walk, GIT_SORT_TOPOLOGICAL, commit_sorting_topo, 2)); - must_pass(test_walk(walk, head, - GIT_SORT_TOPOLOGICAL, - commit_sorting_topo, 2)); - - must_pass(test_walk(walk, head, - GIT_SORT_TIME | GIT_SORT_REVERSE, - commit_sorting_time_reverse, 1)); - - must_pass(test_walk(walk, head, - GIT_SORT_TOPOLOGICAL | GIT_SORT_REVERSE, - commit_sorting_topo_reverse, 2)); + must_pass(test_walk(walk, GIT_SORT_TIME | GIT_SORT_REVERSE, commit_sorting_time_reverse, 1)); + must_pass(test_walk(walk, GIT_SORT_TOPOLOGICAL | GIT_SORT_REVERSE, commit_sorting_topo_reverse, 2)); git_revwalk_free(walk); git_repository_free(repo); END_TEST -BEGIN_TEST(list0, "check that a commit list is properly sorted by time") - - git_revwalk_list list; - git_revwalk_listnode *n; - int i, t; - time_t previous_time; - -#define TEST_SORTED() \ - previous_time = INT_MAX;\ - for (n = list.head; n != NULL; n = n->next) {\ - must_be_true(n->walk_commit->commit_object->committer->when.time <= previous_time);\ - previous_time = n->walk_commit->commit_object->committer->when.time;\ - } - -#define CLEAR_LIST() \ - for (n = list.head; n != NULL; n = n->next) {\ - git_signature_free(n->walk_commit->commit_object->committer);\ - free(n->walk_commit->commit_object);\ - free(n->walk_commit);\ - }\ - git_revwalk_list_clear(&list); - - memset(&list, 0x0, sizeof(git_revwalk_list)); - srand((unsigned int)time(NULL)); - - for (t = 0; t < 20; ++t) { - const int test_size = rand() % 500 + 500; - - /* Purely random sorting test */ - for (i = 0; i < test_size; ++i) { - git_commit *c = git__malloc(sizeof(git_commit)); - git_revwalk_commit *rc = git__malloc(sizeof(git_revwalk_commit)); - - c->committer = git_signature_new("", "", (time_t)rand(), 0); - rc->commit_object = c; - - git_revwalk_list_push_back(&list, rc); - } - - git_revwalk_list_timesort(&list); - TEST_SORTED(); - CLEAR_LIST(); - } - - /* Try to sort list with all dates equal. */ - for (i = 0; i < 200; ++i) { - git_commit *c = git__malloc(sizeof(git_commit)); - git_revwalk_commit *rc = git__malloc(sizeof(git_revwalk_commit)); - - c->committer = git_signature_new("", "", 0, 0); - rc->commit_object = c; - - git_revwalk_list_push_back(&list, rc); - } - - git_revwalk_list_timesort(&list); - TEST_SORTED(); - CLEAR_LIST(); - - /* Try to sort empty list */ - git_revwalk_list_timesort(&list); - TEST_SORTED(); - -END_TEST - BEGIN_SUITE(revwalk) ADD_TEST(walk0); - ADD_TEST(list0); END_SUITE From 58d06cf120eb9bb9247bb807bb105981bb3482a8 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Thu, 10 Mar 2011 01:06:24 +0200 Subject: [PATCH 05/81] Rewrite the Pack backend The new pack backend is an adaptation of the original git.git code in `sha1_file.c`. It's slightly faster than the previous version and severely less memory-hungry. The call-stack of a normal pack backend query has been properly documented in the top of the header for future reference. And by properly I mean with ASCII diagrams 'n shit. --- src/odb_pack.c | 2290 ++++++++++++++++++++++++++---------------------- src/util.h | 2 + 2 files changed, 1246 insertions(+), 1046 deletions(-) diff --git a/src/odb_pack.c b/src/odb_pack.c index 664b00139..3067179be 100644 --- a/src/odb_pack.c +++ b/src/odb_pack.c @@ -33,1082 +33,623 @@ #include "git2/odb_backend.h" -/** First 4 bytes of a pack-*.idx file header. +#define DEFAULT_WINDOW_SIZE \ + (sizeof(void*) >= 8 \ + ? 1 * 1024 * 1024 * 1024 \ + : 32 * 1024 * 1024) + +#define DEFAULT_MAPPED_LIMIT \ + ((1024L * 1024L) * (sizeof(void*) >= 8 ? 8192 : 256)) + +#define PACK_SIGNATURE 0x5041434b /* "PACK" */ +#define PACK_VERSION 2 +#define pack_version_ok(v) ((v) == htonl(2) || (v) == htonl(3)) +struct pack_header { + uint32_t hdr_signature; + uint32_t hdr_version; + uint32_t hdr_entries; +}; + +/* + * The first four bytes of index formats later than version 1 should + * start with this signature, as all older git binaries would find this + * value illegal and abort reading the file. * - * Note this header exists only in idx v2 and later. The idx v1 - * file format does not have a magic sequence at the front, and - * must be detected by the first four bytes *not* being this value - * and the first 8 bytes matching the following expression: + * This is the case because the number of objects in a packfile + * cannot exceed 1,431,660,000 as every object would need at least + * 3 bytes of data and the overall packfile cannot exceed 4 GiB with + * version 1 of the index file due to the offsets limited to 32 bits. + * Clearly the signature exceeds this maximum. * - * uint32_t *fanout = ... the file data at offset 0 ... - * ntohl(fanout[0]) < ntohl(fanout[1]) - * - * The value chosen here for PACK_TOC is such that the above - * cannot be true for an idx v1 file. + * Very old git binaries will also compare the first 4 bytes to the + * next 4 bytes in the index and abort with a "non-monotonic index" + * error if the second 4 byte word is smaller than the first 4 + * byte word. This would be true in the proposed future index + * format as idx_signature would be greater than idx_version. */ -#define PACK_TOC 0xff744f63 /* -1tOc */ +#define PACK_IDX_SIGNATURE 0xff744f63 /* "\377tOc" */ -/** First 4 bytes of a pack-*.pack file header. */ -#define PACK_SIG 0x5041434b /* PACK */ +struct pack_idx_header { + uint32_t idx_signature; + uint32_t idx_version; +}; -#define GIT_PACK_NAME_MAX (5 + 40 + 1) +struct pack_window { + struct pack_window *next; + git_map window_map; + off_t offset; + unsigned int last_used; + unsigned int inuse_cnt; +}; -struct pack_backend; +struct pack_file { + struct pack_window *windows; + off_t pack_size; -typedef struct { - uint32_t n; - unsigned char *oid; - git_off_t offset; - git_off_t size; -} index_entry; + git_map index_map; -typedef struct { /* '.pack' file header */ - uint32_t sig; /* PACK_SIG */ - uint32_t ver; /* pack version */ - uint32_t cnt; /* object count */ -} pack_hdr; + uint32_t num_objects; + uint32_t num_bad_objects; + git_oid *bad_object_sha1; /* array of git_oid */ -typedef struct git_pack { + int index_version; + time_t mtime; + int pack_fd; + unsigned pack_local:1, pack_keep:1; + git_oid sha1; + + /* something like ".git/objects/pack/xxxxx.pack" */ + char pack_name[GIT_FLEX_ARRAY]; /* more */ +}; + +struct pack_entry { + off_t offset; + git_oid sha1; + struct pack_file *p; +}; + +struct pack__dirent { struct pack_backend *backend; - git_lck lock; + int is_pack_local; +}; - /** Functions to access idx_map. */ - int (*idx_search)( - uint32_t *, - struct git_pack *, - const git_oid *); - int (*idx_search_offset)( - uint32_t *, - struct git_pack *, - git_off_t); - int (*idx_get)( - index_entry *, - struct git_pack *, - uint32_t n); - - /** The .idx file, mapped into memory. */ - git_file idx_fd; - git_map idx_map; - uint32_t *im_fanout; - unsigned char *im_oid; - uint32_t *im_crc; - uint32_t *im_offset32; - uint32_t *im_offset64; - uint32_t *im_off_idx; - uint32_t *im_off_next; - - /** Number of objects in this pack. */ - uint32_t obj_cnt; - - /** File descriptor for the .pack file. */ - git_file pack_fd; - - /** Memory map of the pack's contents */ - git_map pack_map; - - /** The size of the .pack file. */ - git_off_t pack_size; - - /** The mtime of the .pack file. */ - time_t pack_mtime; - - /** Number of git_packlist we appear in. */ - unsigned int refcnt; - - /** Number of active users of the idx_map data. */ - unsigned int idxcnt; - unsigned - invalid:1 /* the pack is unable to be read by libgit2 */ - ; - - /** Name of the pack file(s), without extension ("pack-abc"). */ - char pack_name[GIT_PACK_NAME_MAX]; -} git_pack; - -typedef struct { - size_t n_packs; - unsigned int refcnt; - git_pack *packs[GIT_FLEX_ARRAY]; -} git_packlist; - -typedef struct pack_backend { +struct pack_backend { git_odb_backend parent; + git_vector packs; + struct pack_file *last_found; - git_lck lock; - char *objects_dir; - git_packlist *packlist; -} pack_backend; + size_t window_size; /* needs default value */ + + size_t mapped_limit; /* needs default value */ + size_t peak_mapped; + size_t mapped; + + size_t used_ctr; + + unsigned int peak_open_windows; + unsigned int open_windows; + + unsigned int mmap_calls; +}; + +/** + * The wonderful tale of a Packed Object lookup query + * =================================================== + * A riveting and epic story of epicness and ASCII + * art, presented by yours truly, + * Sir Vicent of Marti + * + * + * Chapter 1: Once upon a time... + * Initialization of the Pack Backend + * -------------------------------------------------- + * + * # git_odb_backend_pack + * | Creates the pack backend structure, initializes the + * | callback pointers to our default read() and exist() methods, + * | and tries to preload all the known packfiles in the ODB. + * | + * |-# packfile_load_all + * | Tries to find the `pack` folder, if it exists. ODBs without + * | a pack folder are ignored altogether. If there's a `pack` folder + * | we run a `dirent` callback through every file in the pack folder + * | to find our packfiles. The packfiles are then sorted according + * | to a sorting callback. + * | + * |-# packfile_load__cb + * | | This callback is called from `dirent` with every single file + * | | inside the pack folder. We find the packs by actually locating + * | | their index (ends in ".idx"). From that index, we verify that + * | | the corresponding packfile exists and is valid, and if so, we + * | | add it to the pack list. + * | | + * | |-# packfile_check + * | Make sure that there's a packfile to back this index, and store + * | some very basic information regarding the packfile itself, + * | such as the full path, the size, and the modification time. + * | We don't actually open the packfile to check for internal consistency. + * | + * |-# packfile_sort__cb + * Sort all the preloaded packs according to some specific criteria: + * we prioritize the "newer" packs because it's more likely they + * contain the objects we are looking for, and we prioritize local + * packs over remote ones. + * + * + * + * Chapter 2: To be, or not to be... + * A standard packed `exist` query for an OID + * -------------------------------------------------- + * + * # pack_backend__exists + * | Check if the given SHA1 oid exists in any of the packs + * | that have been loaded for our ODB. + * | + * |-# pack_entry_find + * | Iterate through all the packs that have been preloaded + * | (starting by the pack where the latest object was found) + * | to try to find the OID in one of them. + * | + * |-# pack_entry_find1 + * | Check the index of an individual pack to see if the SHA1 + * | OID can be found. If we can find the offset to that SHA1 + * | inside of the index, that means the object is contained + * | inside of the packfile and we can stop searching. + * | Before returning, we verify that the packfile behing the + * | index we are searching still exists on disk. + * | + * |-# pack_entry_find_offset + * | | Mmap the actual index file to disk if it hasn't been opened + * | | yet, and run a binary search through it to find the OID. + * | | See for specifics + * | | on the Packfile Index format and how do we find entries in it. + * | | + * | |-# pack_index_open + * | | Guess the name of the index based on the full path to the + * | | packfile, open it and verify its contents. Only if the index + * | | has not been opened already. + * | | + * | |-# pack_index_check + * | Mmap the index file and do a quick run through the header + * | to guess the index version (right now we support v1 and v2), + * | and to verify that the size of the index makes sense. + * | + * |-# packfile_open + * See `packfile_open` in Chapter 3 + * + * + * + * Chapter 3: The neverending story... + * A standard packed `lookup` query for an OID + * -------------------------------------------------- + * TODO + * + */ -typedef struct pack_location { - git_pack *ptr; - uint32_t n; -} pack_location; -static int pack_stat(git_pack *p); -static int pack_openidx(git_pack *p); -static void pack_decidx(git_pack *p); -static int read_pack_hdr(pack_hdr *out, git_file fd); -static int check_pack_hdr(git_pack *p); -static int check_pack_sha1(git_pack *p); -static int open_pack(git_pack *p); + +/*********************************************************** + * + * FORWARD DECLARATIONS + * + ***********************************************************/ + +static void pack_window_free_all(struct pack_backend *backend, struct pack_file *p); +static int pack_window_contains(struct pack_window *win, off_t offset); + +static void pack_window_scan_lru(struct pack_file *p, struct pack_file **lru_p, + struct pack_window **lru_w, struct pack_window **lru_l); + +static int pack_window_close_lru( struct pack_backend *backend, + struct pack_file *current, git_file keep_fd); + +static void pack_window_close(struct pack_window **w_cursor); + +static unsigned char *pack_window_open( struct pack_backend *backend, + struct pack_file *p, struct pack_window **w_cursor, off_t offset, + unsigned int *left); + +static int packfile_sort__cb(const void *a_, const void *b_); + +static void pack_index_free(struct pack_file *p); + +static int pack_index_check(const char *path, struct pack_file *p); +static int pack_index_open(struct pack_file *p); + +static struct pack_file *packfile_alloc(int extra); +static int packfile_open(struct pack_file *p); +static int packfile_check(struct pack_file **pack_out, const char *path, int local); +static int packfile_load__cb(void *_data, char *path); +static int packfile_load_all(struct pack_backend *backend, const char *odb_path, int local); + +static off_t nth_packed_object_offset(const struct pack_file *p, uint32_t n); + +static int pack_entry_find_offset(off_t *offset_out, + struct pack_file *p, const git_oid *oid); + +static int pack_entry_find1(struct pack_entry *e, + struct pack_file *p, const git_oid *oid); + +static int pack_entry_find(struct pack_entry *e, + struct pack_backend *backend, const git_oid *oid); + +static off_t get_delta_base(struct pack_backend *backend, + struct pack_file *p, struct pack_window **w_curs, + off_t *curpos, git_otype type, + off_t delta_obj_offset); + +static unsigned long packfile_unpack_header1( + size_t *sizep, + git_otype *type, + const unsigned char *buf, + unsigned long len); + +static int packfile_unpack_header( + size_t *size_p, + git_otype *type_p, + struct pack_backend *backend, + struct pack_file *p, + struct pack_window **w_curs, + off_t *curpos); + +static int packfile_unpack_compressed( + git_rawobj *obj, + struct pack_backend *backend, + struct pack_file *p, + struct pack_window **w_curs, + off_t curpos, + size_t size, + git_otype type); + +static int packfile_unpack_delta( + git_rawobj *obj, + struct pack_backend *backend, + struct pack_file *p, + struct pack_window **w_curs, + off_t curpos, + size_t delta_size, + git_otype delta_type, + off_t obj_offset); + +static int packfile_unpack(git_rawobj *obj, struct pack_backend *backend, + struct pack_file *p, off_t obj_offset); -static int pack_openidx_map(git_pack *p); -static int pack_openidx_v1(git_pack *p); -static int pack_openidx_v2(git_pack *p); - - -GIT_INLINE(uint32_t) decode32(void *b) -{ - return ntohl(*((uint32_t *)b)); -} - -GIT_INLINE(uint64_t) decode64(void *b) -{ - uint32_t *p = b; - return (((uint64_t)ntohl(p[0])) << 32) | ntohl(p[1]); -} /*********************************************************** * - * PACKFILE FUNCTIONS - * - * Locate, open and access the contents of a packfile + * PACK WINDOW MANAGEMENT * ***********************************************************/ -static int pack_stat(git_pack *p) +void pack_window_free_all(struct pack_backend *backend, struct pack_file *p) { - char pb[GIT_PATH_MAX]; - struct stat sb; + while (p->windows) { + struct pack_window *w = p->windows; + assert(w->inuse_cnt == 0); - if (git__fmt(pb, sizeof(pb), "%s/pack/%s.pack", - p->backend->objects_dir, - p->pack_name) < 0) - return GIT_ERROR; + backend->mapped -= w->window_map.len; + backend->open_windows--; - if (gitfo_stat(pb, &sb) || !S_ISREG(sb.st_mode)) - return GIT_ERROR; + gitfo_free_map(&w->window_map); - if (sb.st_size < (3 * 4 + GIT_OID_RAWSZ)) - return GIT_ERROR; - - p->pack_size = sb.st_size; - p->pack_mtime = sb.st_mtime; - - return GIT_SUCCESS; -} - -static int pack_openidx(git_pack *p) -{ - gitlck_lock(&p->lock); - - if (p->invalid) { - gitlck_unlock(&p->lock); - return GIT_ERROR; + p->windows = w->next; + free(w); } +} - if (++p->idxcnt == 1 && !p->idx_search) { - int status, version; - uint32_t *data; +GIT_INLINE(int) pack_window_contains(struct pack_window *win, off_t offset) +{ + /* We must promise at least 20 bytes (one hash) after the + * offset is available from this window, otherwise the offset + * is not actually in this window and a different window (which + * has that one hash excess) must be used. This is to support + * the object header and delta base parsing routines below. + */ + off_t win_off = win->offset; + return win_off <= offset + && (offset + 20) <= (off_t)(win_off + win->window_map.len); +} - if (pack_stat(p) || pack_openidx_map(p)) { - p->invalid = 1; - p->idxcnt--; - gitlck_unlock(&p->lock); - return GIT_ERROR; - } - data = p->idx_map.data; - status = GIT_SUCCESS; - version = 1; - - if (decode32(&data[0]) == PACK_TOC) - version = decode32(&data[1]); - - switch (version) { - case 1: - status = pack_openidx_v1(p); - break; - case 2: - status = pack_openidx_v2(p); - break; - default: - status = GIT_ERROR; - } - - if (status != GIT_SUCCESS) { - gitfo_free_map(&p->idx_map); - p->invalid = 1; - p->idxcnt--; - gitlck_unlock(&p->lock); - return status; +static void pack_window_scan_lru( + struct pack_file *p, + struct pack_file **lru_p, + struct pack_window **lru_w, + struct pack_window **lru_l) +{ + struct pack_window *w, *w_l; + + for (w_l = NULL, w = p->windows; w; w = w->next) { + if (!w->inuse_cnt) { + if (!*lru_w || w->last_used < (*lru_w)->last_used) { + *lru_p = p; + *lru_w = w; + *lru_l = w_l; + } } + w_l = w; } - - gitlck_unlock(&p->lock); - return GIT_SUCCESS; } -static void pack_decidx(git_pack *p) +static int pack_window_close_lru( + struct pack_backend *backend, + struct pack_file *current, + git_file keep_fd) { - gitlck_lock(&p->lock); - p->idxcnt--; - gitlck_unlock(&p->lock); -} + struct pack_file *lru_p = NULL; + struct pack_window *lru_w = NULL, *lru_l = NULL; + size_t i; -static int read_pack_hdr(pack_hdr *out, git_file fd) -{ - pack_hdr hdr; + if (current) + pack_window_scan_lru(current, &lru_p, &lru_w, &lru_l); - if (gitfo_read(fd, &hdr, sizeof(hdr))) - return GIT_ERROR; + for (i = 0; i < backend->packs.length; ++i) + pack_window_scan_lru(git_vector_get(&backend->packs, i), &lru_p, &lru_w, &lru_l); - out->sig = decode32(&hdr.sig); - out->ver = decode32(&hdr.ver); - out->cnt = decode32(&hdr.cnt); + if (lru_p) { + backend->mapped -= lru_w->window_map.len; + gitfo_free_map(&lru_w->window_map); - return GIT_SUCCESS; -} + if (lru_l) + lru_l->next = lru_w->next; + else { + lru_p->windows = lru_w->next; + if (!lru_p->windows && lru_p->pack_fd != keep_fd) { + gitfo_close(lru_p->pack_fd); + lru_p->pack_fd = -1; + } + } -static int check_pack_hdr(git_pack *p) -{ - pack_hdr hdr; - - if (read_pack_hdr(&hdr, p->pack_fd)) - return GIT_ERROR; - - if (hdr.sig != PACK_SIG - || (hdr.ver != 2 && hdr.ver != 3) - || hdr.cnt != p->obj_cnt) - return GIT_ERROR; - - return GIT_SUCCESS; -} - -static int check_pack_sha1(git_pack *p) -{ - unsigned char *data = p->idx_map.data; - git_off_t pack_sha1_off = p->pack_size - GIT_OID_RAWSZ; - size_t idx_pack_sha1_off = p->idx_map.len - 2 * GIT_OID_RAWSZ; - git_oid pack_id, idx_pack_id; - - if (gitfo_lseek(p->pack_fd, pack_sha1_off, SEEK_SET) == -1) - return GIT_ERROR; - - if (gitfo_read(p->pack_fd, pack_id.id, sizeof(pack_id.id))) - return GIT_ERROR; - - git_oid_mkraw(&idx_pack_id, data + idx_pack_sha1_off); - - if (git_oid_cmp(&pack_id, &idx_pack_id)) - return GIT_ERROR; - - return GIT_SUCCESS; -} - -static int open_pack(git_pack *p) -{ - char pb[GIT_PATH_MAX]; - struct stat sb; - - if (p->pack_fd != -1) + free(lru_w); + backend->open_windows--; return GIT_SUCCESS; + } - if (git__fmt(pb, sizeof(pb), "%s/pack/%s.pack", - p->backend->objects_dir, - p->pack_name) < 0) - return GIT_ERROR; - - if (pack_openidx(p)) - return GIT_ERROR; - - if ((p->pack_fd = gitfo_open(pb, O_RDONLY)) < 0) - goto error_cleanup; - - if (gitfo_fstat(p->pack_fd, &sb) - || !S_ISREG(sb.st_mode) || p->pack_size != sb.st_size - || check_pack_hdr(p) || check_pack_sha1(p)) - goto error_cleanup; - - if (!git__is_sizet(p->pack_size) || - gitfo_map_ro(&p->pack_map, p->pack_fd, 0, (size_t)p->pack_size) < 0) - goto error_cleanup; - - pack_decidx(p); - return GIT_SUCCESS; - -error_cleanup: - gitfo_close(p->pack_fd); - p->pack_fd = -1; - pack_decidx(p); return GIT_ERROR; } -static void pack_dec(git_pack *p) +static void pack_window_close(struct pack_window **w_cursor) { - int need_free; - - gitlck_lock(&p->lock); - need_free = !--p->refcnt; - gitlck_unlock(&p->lock); - - if (need_free) { - if (p->idx_search) { - gitfo_free_map(&p->idx_map); - gitfo_close(p->idx_fd); - free(p->im_fanout); - free(p->im_off_idx); - free(p->im_off_next); - if (p->pack_fd != -1) { - gitfo_close(p->pack_fd); - gitfo_free_map(&p->pack_map); - } - } - - gitlck_free(&p->lock); - free(p); + struct pack_window *w = *w_cursor; + if (w) { + w->inuse_cnt--; + *w_cursor = NULL; } } -static void packlist_dec(pack_backend *backend, git_packlist *pl) +static unsigned char *pack_window_open( + struct pack_backend *backend, + struct pack_file *p, + struct pack_window **w_cursor, + off_t offset, + unsigned int *left) { - int need_free; + struct pack_window *win = *w_cursor; - assert(backend && pl); - - gitlck_lock(&backend->lock); - need_free = !--pl->refcnt; - gitlck_unlock(&backend->lock); - - if (need_free) { - size_t j; - for (j = 0; j < pl->n_packs; j++) - pack_dec(pl->packs[j]); - free(pl); - } -} - -static git_pack *alloc_pack(const char *pack_name) -{ - git_pack *p = git__calloc(1, sizeof(*p)); - if (!p) + if (p->pack_fd == -1 && packfile_open(p) < GIT_SUCCESS) return NULL; - gitlck_init(&p->lock); - strcpy(p->pack_name, pack_name); - p->refcnt = 1; - p->pack_fd = -1; - return p; -} - -struct scanned_pack { - struct scanned_pack *next; - git_pack *pack; -}; - -static int scan_one_pack(void *state, char *name) -{ - struct scanned_pack **ret = state, *r; - char *s = strrchr(name, '/'), *d; - - if (git__prefixcmp(s + 1, "pack-") - || git__suffixcmp(s, ".pack") - || strlen(s + 1) != GIT_PACK_NAME_MAX + 4) - return 0; - - d = strrchr(s + 1, '.'); - strcpy(d + 1, "idx"); /* "pack-abc.pack" -> "pack-abc.idx" */ - if (gitfo_exists(name)) - return 0; - - if ((r = git__malloc(sizeof(*r))) == NULL) - return GIT_ERROR; - - *d = '\0'; /* "pack-abc.pack" -_> "pack-abc" */ - if ((r->pack = alloc_pack(s + 1)) == NULL) { - free(r); - return GIT_ERROR; - } - - r->next = *ret; - *ret = r; - return 0; -} - -static git_packlist *scan_packs(pack_backend *backend) -{ - char pb[GIT_PATH_MAX]; - struct scanned_pack *state = NULL, *c; - size_t cnt; - git_packlist *new_list; - - if (git__fmt(pb, sizeof(pb), "%s/pack", backend->objects_dir) < 0) - return NULL; - gitfo_dirent(pb, sizeof(pb), scan_one_pack, &state); - - /* TODO - merge old entries into the new array */ - for (cnt = 0, c = state; c; c = c->next) - cnt++; - new_list = git__malloc(sizeof(*new_list) - + (sizeof(new_list->packs[0]) * cnt)); - if (!new_list) - goto fail; - - for (cnt = 0, c = state; c; ) { - struct scanned_pack *n = c->next; - c->pack->backend = backend; - new_list->packs[cnt++] = c->pack; - free(c); - c = n; - } - new_list->n_packs = cnt; - new_list->refcnt = 2; - backend->packlist = new_list; - return new_list; - -fail: - while (state) { - struct scanned_pack *n = state->next; - pack_dec(state->pack); - free(state); - state = n; - } - return NULL; -} - -static git_packlist *packlist_get(pack_backend *backend) -{ - git_packlist *pl; - - gitlck_lock(&backend->lock); - if ((pl = backend->packlist) != NULL) - pl->refcnt++; - else - pl = scan_packs(backend); - gitlck_unlock(&backend->lock); - return pl; -} - -static int locate_packfile(pack_location *location, pack_backend *backend, const git_oid *id) -{ - git_packlist *pl = packlist_get(backend); - size_t j; - - if (!pl) - return GIT_ENOTFOUND; - - for (j = 0; j < pl->n_packs; j++) { - - git_pack *pack = pl->packs[j]; - uint32_t pos; - int res; - - if (pack_openidx(pack)) - continue; - - res = pack->idx_search(&pos, pack, id); - pack_decidx(pack); - - if (!res) { - packlist_dec(backend, pl); - - location->ptr = pack; - location->n = pos; - - return GIT_SUCCESS; - } - - } - - packlist_dec(backend, pl); - return GIT_ENOTFOUND; -} - - - - - - - - - - - - - -/*********************************************************** - * - * PACKFILE INDEX FUNCTIONS - * - * Get index formation for packfile indexes v1 and v2 - * - ***********************************************************/ - -static int pack_openidx_map(git_pack *p) -{ - char pb[GIT_PATH_MAX]; - git_off_t len; - - if (git__fmt(pb, sizeof(pb), "%s/pack/%s.idx", - p->backend->objects_dir, - p->pack_name) < 0) - return GIT_ERROR; - - if ((p->idx_fd = gitfo_open(pb, O_RDONLY)) < 0) - return GIT_ERROR; - - if ((len = gitfo_size(p->idx_fd)) < 0 - || !git__is_sizet(len) - || gitfo_map_ro(&p->idx_map, p->idx_fd, 0, (size_t)len)) { - gitfo_close(p->idx_fd); - return GIT_ERROR; - } - - return GIT_SUCCESS; -} - -typedef struct { - git_off_t offset; - uint32_t n; -} offset_idx_info; - -static int cmp_offset_idx_info(const void *lhs, const void *rhs) -{ - const offset_idx_info *a = lhs; - const offset_idx_info *b = rhs; - return (a->offset < b->offset) ? -1 : (a->offset > b->offset) ? 1 : 0; -} - -static int make_offset_index(git_pack *p, offset_idx_info *data) -{ - git_off_t min_off = 3 * 4, max_off = p->pack_size - GIT_OID_RAWSZ; - uint32_t *idx, *next; - uint32_t j; - - qsort(data, p->obj_cnt, sizeof(*data), cmp_offset_idx_info); - - if (data[0].offset < min_off || data[p->obj_cnt].offset > max_off) - return GIT_ERROR; - - if ((idx = git__malloc(sizeof(*idx) * (p->obj_cnt+1))) == NULL) - return GIT_ERROR; - if ((next = git__malloc(sizeof(*next) * p->obj_cnt)) == NULL) { - free(idx); - return GIT_ERROR; - } - - for (j = 0; j < p->obj_cnt+1; j++) - idx[j] = data[j].n; - - for (j = 0; j < p->obj_cnt; j++) { - assert(idx[j] < p->obj_cnt); - assert(idx[j+1] < p->obj_cnt+1); - - next[idx[j]] = idx[j+1]; - } - - p->im_off_idx = idx; - p->im_off_next = next; - return GIT_SUCCESS; -} - -static int idxv1_search(uint32_t *out, git_pack *p, const git_oid *id) -{ - unsigned char *data = p->im_oid; - uint32_t lo = id->id[0] ? p->im_fanout[id->id[0] - 1] : 0; - uint32_t hi = p->im_fanout[id->id[0]]; - - do { - uint32_t mid = (lo + hi) >> 1; - uint32_t pos = 24 * mid; - int cmp = memcmp(id->id, data + pos + 4, 20); - if (cmp < 0) - hi = mid; - else if (!cmp) { - *out = mid; - return GIT_SUCCESS; - } else - lo = mid + 1; - } while (lo < hi); - return GIT_ENOTFOUND; -} - -static int idxv1_search_offset(uint32_t *out, git_pack *p, git_off_t offset) -{ - if (offset > 0 && offset < (p->pack_size - GIT_OID_RAWSZ)) { - uint32_t lo = 0, hi = p->obj_cnt+1; - unsigned char *data = p->im_oid; - uint32_t *idx = p->im_off_idx; - do { - uint32_t mid = (lo + hi) >> 1; - uint32_t n = idx[mid]; - uint32_t pos = n * (GIT_OID_RAWSZ + 4); - git_off_t here = decode32(data + pos); - if (offset < here) - hi = mid; - else if (offset == here) { - *out = n; - return GIT_SUCCESS; - } else - lo = mid + 1; - } while (lo < hi); - } - return GIT_ENOTFOUND; -} - -static int idxv1_get(index_entry *e, git_pack *p, uint32_t n) -{ - unsigned char *data = p->im_oid; - uint32_t *next = p->im_off_next; - - if (n < p->obj_cnt) { - uint32_t pos = n * (GIT_OID_RAWSZ + 4); - git_off_t next_off = p->pack_size - GIT_OID_RAWSZ; - e->n = n; - e->oid = data + pos + 4; - e->offset = decode32(data + pos); - if (next[n] < p->obj_cnt) { - pos = next[n] * (GIT_OID_RAWSZ + 4); - next_off = decode32(data + pos); - } - e->size = next_off - e->offset; - return GIT_SUCCESS; - } - return GIT_ENOTFOUND; -} - -static int pack_openidx_v1(git_pack *p) -{ - uint32_t *src_fanout = p->idx_map.data; - uint32_t *im_fanout; - offset_idx_info *info; - size_t expsz; - uint32_t j; - - - if ((im_fanout = git__malloc(sizeof(*im_fanout) * 256)) == NULL) - return GIT_ERROR; - - im_fanout[0] = decode32(&src_fanout[0]); - for (j = 1; j < 256; j++) { - im_fanout[j] = decode32(&src_fanout[j]); - if (im_fanout[j] < im_fanout[j - 1]) { - free(im_fanout); - return GIT_ERROR; - } - } - p->obj_cnt = im_fanout[255]; - - expsz = 4 * 256 + 24 * p->obj_cnt + 2 * 20; - if (expsz != p->idx_map.len) { - free(im_fanout); - return GIT_ERROR; - } - - p->idx_search = idxv1_search; - p->idx_search_offset = idxv1_search_offset; - p->idx_get = idxv1_get; - p->im_fanout = im_fanout; - p->im_oid = (unsigned char *)(src_fanout + 256); - - if ((info = git__malloc(sizeof(*info) * (p->obj_cnt+1))) == NULL) { - free(im_fanout); - return GIT_ERROR; - } - - for (j = 0; j < p->obj_cnt; j++) { - uint32_t pos = j * (GIT_OID_RAWSZ + 4); - info[j].offset = decode32(p->im_oid + pos); - info[j].n = j; - } - info[p->obj_cnt].offset = p->pack_size - GIT_OID_RAWSZ; - info[p->obj_cnt].n = p->obj_cnt; - - if (make_offset_index(p, info)) { - free(im_fanout); - free(info); - return GIT_ERROR; - } - free(info); - - return GIT_SUCCESS; -} - -static int idxv2_search(uint32_t *out, git_pack *p, const git_oid *id) -{ - unsigned char *data = p->im_oid; - uint32_t lo = id->id[0] ? p->im_fanout[id->id[0] - 1] : 0; - uint32_t hi = p->im_fanout[id->id[0]]; - - do { - uint32_t mid = (lo + hi) >> 1; - uint32_t pos = 20 * mid; - int cmp = memcmp(id->id, data + pos, 20); - if (cmp < 0) - hi = mid; - else if (!cmp) { - *out = mid; - return GIT_SUCCESS; - } else - lo = mid + 1; - } while (lo < hi); - return GIT_ENOTFOUND; -} - -static int idxv2_search_offset(uint32_t *out, git_pack *p, git_off_t offset) -{ - if (offset > 0 && offset < (p->pack_size - GIT_OID_RAWSZ)) { - uint32_t lo = 0, hi = p->obj_cnt+1; - uint32_t *idx = p->im_off_idx; - do { - uint32_t mid = (lo + hi) >> 1; - uint32_t n = idx[mid]; - uint32_t o32 = decode32(p->im_offset32 + n); - git_off_t here = o32; - - if (o32 & 0x80000000) { - uint32_t o64_idx = (o32 & ~0x80000000); - here = decode64(p->im_offset64 + 2*o64_idx); - } - - if (offset < here) - hi = mid; - else if (offset == here) { - *out = n; - return GIT_SUCCESS; - } else - lo = mid + 1; - } while (lo < hi); - } - return GIT_ENOTFOUND; -} - -static int idxv2_get(index_entry *e, git_pack *p, uint32_t n) -{ - unsigned char *data = p->im_oid; - uint32_t *next = p->im_off_next; - - if (n < p->obj_cnt) { - uint32_t o32 = decode32(p->im_offset32 + n); - git_off_t next_off = p->pack_size - GIT_OID_RAWSZ; - e->n = n; - e->oid = data + n * GIT_OID_RAWSZ; - e->offset = o32; - if (o32 & 0x80000000) { - uint32_t o64_idx = (o32 & ~0x80000000); - e->offset = decode64(p->im_offset64 + 2*o64_idx); - } - if (next[n] < p->obj_cnt) { - o32 = decode32(p->im_offset32 + next[n]); - next_off = o32; - if (o32 & 0x80000000) { - uint32_t o64_idx = (o32 & ~0x80000000); - next_off = decode64(p->im_offset64 + 2*o64_idx); - } - } - e->size = next_off - e->offset; - return GIT_SUCCESS; - } - return GIT_ENOTFOUND; -} - -static int pack_openidx_v2(git_pack *p) -{ - unsigned char *data = p->idx_map.data; - uint32_t *src_fanout = (uint32_t *)(data + 8); - uint32_t *im_fanout; - offset_idx_info *info; - size_t sz, o64_sz, o64_len; - uint32_t j; - - if ((im_fanout = git__malloc(sizeof(*im_fanout) * 256)) == NULL) - return GIT_ERROR; - - im_fanout[0] = decode32(&src_fanout[0]); - for (j = 1; j < 256; j++) { - im_fanout[j] = decode32(&src_fanout[j]); - if (im_fanout[j] < im_fanout[j - 1]) { - free(im_fanout); - return GIT_ERROR; - } - } - p->obj_cnt = im_fanout[255]; - - /* minimum size of .idx file (with empty 64-bit offsets table): */ - sz = 4 + 4 + 256 * 4 + p->obj_cnt * (20 + 4 + 4) + 2 * 20; - if (p->idx_map.len < sz) { - free(im_fanout); - return GIT_ERROR; - } - - p->idx_search = idxv2_search; - p->idx_search_offset = idxv2_search_offset; - p->idx_get = idxv2_get; - p->im_fanout = im_fanout; - p->im_oid = (unsigned char *)(src_fanout + 256); - p->im_crc = (uint32_t *)(p->im_oid + 20 * p->obj_cnt); - p->im_offset32 = p->im_crc + p->obj_cnt; - p->im_offset64 = p->im_offset32 + p->obj_cnt; - - if ((info = git__malloc(sizeof(*info) * (p->obj_cnt+1))) == NULL) { - free(im_fanout); - return GIT_ERROR; - } - - /* check 64-bit offset table index values are within bounds */ - o64_sz = p->idx_map.len - sz; - o64_len = o64_sz / 8; - for (j = 0; j < p->obj_cnt; j++) { - uint32_t o32 = decode32(p->im_offset32 + j); - git_off_t offset = o32; - if (o32 & 0x80000000) { - uint32_t o64_idx = (o32 & ~0x80000000); - if (o64_idx >= o64_len) { - free(im_fanout); - free(info); - return GIT_ERROR; - } - offset = decode64(p->im_offset64 + 2*o64_idx); - } - info[j].offset = offset; - info[j].n = j; - } - info[p->obj_cnt].offset = p->pack_size - GIT_OID_RAWSZ; - info[p->obj_cnt].n = p->obj_cnt; - - if (make_offset_index(p, info)) { - free(im_fanout); - free(info); - return GIT_ERROR; - } - free(info); - - return GIT_SUCCESS; -} - - - - - - - - -/*********************************************************** - * - * PACKFILE READING FUNCTIONS - * - * Read the contents of a packfile - * - ***********************************************************/ - - -static int unpack_object(git_rawobj *out, git_pack *p, index_entry *e); - -static int unpack_object_delta(git_rawobj *out, git_pack *p, - index_entry *base_entry, - uint8_t *delta_buffer, - size_t delta_deflated_size, - size_t delta_inflated_size) -{ - int res = 0; - uint8_t *delta = NULL; - git_rawobj base_obj; - - base_obj.data = NULL; - base_obj.type = GIT_OBJ_BAD; - base_obj.len = 0; - - if ((res = unpack_object(&base_obj, p, base_entry)) < 0) - goto cleanup; - - delta = git__malloc(delta_inflated_size + 1); - - if ((res = git_odb__inflate_buffer(delta_buffer, delta_deflated_size, - delta, delta_inflated_size)) < 0) - goto cleanup; - - res = git__delta_apply(out, base_obj.data, base_obj.len, delta, delta_inflated_size); - - out->type = base_obj.type; - -cleanup: - free(delta); - git_rawobj_close(&base_obj); - return res; -} - -static int unpack_object(git_rawobj *out, git_pack *p, index_entry *e) -{ - git_otype object_type; - size_t inflated_size, deflated_size, shift; - uint8_t *buffer, byte; - - assert(out && p && e && git__is_sizet(e->size)); - - if (open_pack(p)) - return GIT_ERROR; - - buffer = (uint8_t *)p->pack_map.data + e->offset; - deflated_size = (size_t)e->size; - - if (deflated_size == 0) - deflated_size = (size_t)(p->pack_size - e->offset); - - byte = *buffer++ & 0xFF; - deflated_size--; - object_type = (byte >> 4) & 0x7; - inflated_size = byte & 0xF; - shift = 4; - - while (byte & 0x80) { - byte = *buffer++ & 0xFF; - deflated_size--; - inflated_size += (byte & 0x7F) << shift; - shift += 7; - } - - switch (object_type) { - case GIT_OBJ_COMMIT: - case GIT_OBJ_TREE: - case GIT_OBJ_BLOB: - case GIT_OBJ_TAG: { - - /* Handle a normal zlib stream */ - out->len = inflated_size; - out->type = object_type; - out->data = git__malloc(inflated_size + 1); - - if (git_odb__inflate_buffer(buffer, deflated_size, out->data, out->len) < 0) { - free(out->data); - out->data = NULL; - return GIT_ERROR; - } - - return GIT_SUCCESS; - } - - case GIT_OBJ_OFS_DELTA: { - - git_off_t delta_offset; - index_entry entry; - - byte = *buffer++ & 0xFF; - delta_offset = byte & 0x7F; - - while (byte & 0x80) { - delta_offset += 1; - byte = *buffer++ & 0xFF; - delta_offset <<= 7; - delta_offset += (byte & 0x7F); - } - - entry.n = 0; - entry.oid = NULL; - entry.offset = e->offset - delta_offset; - entry.size = 0; - - if (unpack_object_delta(out, p, &entry, - buffer, deflated_size, inflated_size) < 0) - return GIT_ERROR; - - return GIT_SUCCESS; - } - - case GIT_OBJ_REF_DELTA: { - - git_oid base_id; - uint32_t n; - index_entry entry; - int res = GIT_ERROR; - - git_oid_mkraw(&base_id, buffer); - - if (!p->idx_search(&n, p, &base_id) && - !p->idx_get(&entry, p, n)) { - - res = unpack_object_delta(out, p, &entry, - buffer + GIT_OID_RAWSZ, deflated_size, inflated_size); - } - - return res; - } - - default: - return GIT_EOBJCORRUPTED; - } -} - -static int read_packed(git_rawobj *out, const pack_location *loc) -{ - index_entry e; - int res; - - assert(out && loc); - - if (pack_openidx(loc->ptr) < 0) - return GIT_EPACKCORRUPTED; - - res = loc->ptr->idx_get(&e, loc->ptr, loc->n); - - if (!res) - res = unpack_object(out, loc->ptr, &e); - - pack_decidx(loc->ptr); - - return res; -} - -static int read_header_packed(git_rawobj *out, const pack_location *loc) -{ - git_pack *pack; - index_entry e; - int error = GIT_SUCCESS, shift; - uint8_t *buffer, byte; - - assert(out && loc); - - pack = loc->ptr; - - if (pack_openidx(pack)) - return GIT_EPACKCORRUPTED; - - if (pack->idx_get(&e, pack, loc->n) < 0 || - open_pack(pack) < 0) { - error = GIT_ENOTFOUND; - goto cleanup; - } - - buffer = (uint8_t *)pack->pack_map.data + e.offset; - - byte = *buffer++ & 0xFF; - out->type = (byte >> 4) & 0x7; - out->len = byte & 0xF; - shift = 4; - - while (byte & 0x80) { - byte = *buffer++ & 0xFF; - out->len += (byte & 0x7F) << shift; - shift += 7; - } - - /* - * FIXME: if the object is not packed as a whole, - * we need to do a full load and apply the deltas before - * being able to read the header. - * - * I don't think there are any workarounds for this.' + /* Since packfiles end in a hash of their content and it's + * pointless to ask for an offset into the middle of that + * hash, and the pack_window_contains function above wouldn't match + * don't allow an offset too close to the end of the file. */ + if (offset > (p->pack_size - 20)) + return NULL; - if (out->type == GIT_OBJ_OFS_DELTA || out->type == GIT_OBJ_REF_DELTA) { - error = unpack_object(out, pack, &e); - git_rawobj_close(out); + if (!win || !pack_window_contains(win, offset)) { + + if (win) + win->inuse_cnt--; + + for (win = p->windows; win; win = win->next) { + if (pack_window_contains(win, offset)) + break; + } + + if (!win) { + size_t window_align = backend->window_size / 2; + size_t len; + + win = git__calloc(1, sizeof(*win)); + win->offset = (offset / window_align) * window_align; + + len = (size_t)(p->pack_size - win->offset); + if (len > backend->window_size) + len = backend->window_size; + + backend->mapped += len; + + while (backend->mapped_limit < backend->mapped && + pack_window_close_lru(backend, p, p->pack_fd) == GIT_SUCCESS) {} + + if (gitfo_map_ro(&win->window_map, p->pack_fd, + win->offset, len) < GIT_SUCCESS) + return NULL; + + backend->mmap_calls++; + backend->open_windows++; + + if (backend->mapped > backend->peak_mapped) + backend->peak_mapped = backend->mapped; + + if (backend->open_windows > backend->peak_open_windows) + backend->peak_open_windows = backend->open_windows; + + win->next = p->windows; + p->windows = win; + } } -cleanup: - pack_decidx(loc->ptr); + if (win != *w_cursor) { + win->last_used = backend->used_ctr++; + win->inuse_cnt++; + *w_cursor = win; + } + + offset -= win->offset; + assert(git__is_sizet(offset)); + + if (left) + *left = win->window_map.len - (size_t)offset; + + return (unsigned char *)win->window_map.data + offset; +} + + + + + + + +/*********************************************************** + * + * PACK INDEX METHODS + * + ***********************************************************/ + +static void pack_index_free(struct pack_file *p) +{ + if (p->index_map.data) { + gitfo_free_map(&p->index_map); + p->index_map.data = NULL; + } +} + +static int pack_index_check(const char *path, struct pack_file *p) +{ + struct pack_idx_header *hdr; + uint32_t version, nr, i, *index; + + void *idx_map; + size_t idx_size; + + struct stat st; + + /* TODO: properly open the file without access time */ + git_file fd = gitfo_open(path, O_RDONLY /*| O_NOATIME */); + + int error; + + if (fd < 0) + return GIT_EOSERR; + + if (gitfo_fstat(fd, &st) < GIT_SUCCESS) { + gitfo_close(fd); + return GIT_EOSERR; + } + + if (!git__is_sizet(st.st_size)) + return GIT_ENOMEM; + + idx_size = (size_t)st.st_size; + + if (idx_size < 4 * 256 + 20 + 20) { + gitfo_close(fd); + return GIT_EOBJCORRUPTED; + } + + error = gitfo_map_ro(&p->index_map, fd, 0, idx_size); + gitfo_close(fd); + + if (error < GIT_SUCCESS) + return error; + + hdr = idx_map = p->index_map.data; + + if (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) { + version = ntohl(hdr->idx_version); + + if (version < 2 || version > 2) { + gitfo_free_map(&p->index_map); + return GIT_EOBJCORRUPTED; /* unsupported index version */ + } + + } else + version = 1; + + nr = 0; + index = idx_map; + + if (version > 1) + index += 2; /* skip index header */ + + for (i = 0; i < 256; i++) { + uint32_t n = ntohl(index[i]); + if (n < nr) { + gitfo_free_map(&p->index_map); + return GIT_EOBJCORRUPTED; /* non-monotonic index */ + } + nr = n; + } + + if (version == 1) { + /* + * Total size: + * - 256 index entries 4 bytes each + * - 24-byte entries * nr (20-byte sha1 + 4-byte offset) + * - 20-byte SHA1 of the packfile + * - 20-byte SHA1 file checksum + */ + if (idx_size != 4*256 + nr * 24 + 20 + 20) { + gitfo_free_map(&p->index_map); + return GIT_EOBJCORRUPTED; + } + } else if (version == 2) { + /* + * Minimum size: + * - 8 bytes of header + * - 256 index entries 4 bytes each + * - 20-byte sha1 entry * nr + * - 4-byte crc entry * nr + * - 4-byte offset entry * nr + * - 20-byte SHA1 of the packfile + * - 20-byte SHA1 file checksum + * And after the 4-byte offset table might be a + * variable sized table containing 8-byte entries + * for offsets larger than 2^31. + */ + unsigned long min_size = 8 + 4*256 + nr*(20 + 4 + 4) + 20 + 20; + unsigned long max_size = min_size; + + if (nr) + max_size += (nr - 1)*8; + + if (idx_size < min_size || idx_size > max_size) { + gitfo_free_map(&p->index_map); + return GIT_EOBJCORRUPTED; + } + + /* Make sure that off_t is big enough to access the whole pack... + * Is this an issue in libgit2? It shouldn't. */ + if (idx_size != min_size && (sizeof(off_t) <= 4)) { + gitfo_free_map(&p->index_map); + return GIT_EOSERR; + } + } + + p->index_version = version; + p->num_objects = nr; + return GIT_SUCCESS; +} + +static int pack_index_open(struct pack_file *p) +{ + char *idx_name; + int error; + + if (p->index_map.data) + return GIT_SUCCESS; + + idx_name = git__strdup(p->pack_name); + strcpy(idx_name + strlen(idx_name) - STRLEN(".pack"), ".idx"); + + error = pack_index_check(idx_name, p); + free(idx_name); + return error; } @@ -1118,6 +659,662 @@ cleanup: + + +/*********************************************************** + * + * PACKFILE METHODS + * + ***********************************************************/ + +static int packfile_sort__cb(const void *a_, const void *b_) +{ + struct pack_file *a = *((struct pack_file **)a_); + struct pack_file *b = *((struct pack_file **)b_); + int st; + + /* + * Local packs tend to contain objects specific to our + * variant of the project than remote ones. In addition, + * remote ones could be on a network mounted filesystem. + * Favor local ones for these reasons. + */ + st = a->pack_local - b->pack_local; + if (st) + return -st; + + /* + * Younger packs tend to contain more recent objects, + * and more recent objects tend to get accessed more + * often. + */ + if (a->mtime < b->mtime) + return 1; + else if (a->mtime == b->mtime) + return 0; + + return -1; +} + +static struct pack_file *packfile_alloc(int extra) +{ + struct pack_file *p = git__malloc(sizeof(*p) + extra); + memset(p, 0, sizeof(*p)); + p->pack_fd = -1; + return p; +} + + +static void packfile_free(struct pack_backend *backend, struct pack_file *p) +{ + assert(p); + + /* clear_delta_base_cache(); */ + pack_window_free_all(backend, p); + + if (p->pack_fd != -1) + gitfo_close(p->pack_fd); + + pack_index_free(p); + + free(p->bad_object_sha1); + free(p); +} + +static int packfile_open(struct pack_file *p) +{ + struct stat st; + struct pack_header hdr; + git_oid sha1; + unsigned char *idx_sha1; + + if (!p->index_map.data && pack_index_open(p) < GIT_SUCCESS) + return GIT_ENOTFOUND; + + /* TODO: open with noatime */ + p->pack_fd = gitfo_open(p->pack_name, O_RDONLY); + if (p->pack_fd < 0 || gitfo_fstat(p->pack_fd, &st) < GIT_SUCCESS) + return GIT_EOSERR; + + /* If we created the struct before we had the pack we lack size. */ + if (!p->pack_size) { + if (!S_ISREG(st.st_mode)) + goto cleanup; + p->pack_size = (off_t)st.st_size; + } else if (p->pack_size != st.st_size) + goto cleanup; + +#if 0 + /* We leave these file descriptors open with sliding mmap; + * there is no point keeping them open across exec(), though. + */ + fd_flag = fcntl(p->pack_fd, F_GETFD, 0); + if (fd_flag < 0) + return error("cannot determine file descriptor flags"); + + fd_flag |= FD_CLOEXEC; + if (fcntl(p->pack_fd, F_SETFD, fd_flag) == -1) + return GIT_EOSERR; +#endif + + /* Verify we recognize this pack file format. */ + if (gitfo_read(p->pack_fd, &hdr, sizeof(hdr)) < GIT_SUCCESS) + goto cleanup; + + if (hdr.hdr_signature != htonl(PACK_SIGNATURE)) + goto cleanup; + + if (!pack_version_ok(hdr.hdr_version)) + goto cleanup; + + /* Verify the pack matches its index. */ + if (p->num_objects != ntohl(hdr.hdr_entries)) + goto cleanup; + + if (gitfo_lseek(p->pack_fd, p->pack_size - GIT_OID_RAWSZ, SEEK_SET) == -1) + goto cleanup; + + if (gitfo_read(p->pack_fd, sha1.id, GIT_OID_RAWSZ) < GIT_SUCCESS) + goto cleanup; + + idx_sha1 = ((unsigned char *)p->index_map.data) + p->index_map.len - 40; + + if (git_oid_cmp(&sha1, (git_oid *)idx_sha1) != 0) + goto cleanup; + + return GIT_SUCCESS; + +cleanup: + gitfo_close(p->pack_fd); + p->pack_fd = -1; + return GIT_EPACKCORRUPTED; +} + +static int packfile_check(struct pack_file **pack_out, const char *path, int local) +{ + struct stat st; + struct pack_file *p; + size_t path_len; + + *pack_out = NULL; + path_len = strlen(path); + p = packfile_alloc(path_len + 2); + + /* + * Make sure a corresponding .pack file exists and that + * the index looks sane. + */ + path_len -= STRLEN(".idx"); + if (path_len < 1) { + free(p); + return GIT_ENOTFOUND; + } + + memcpy(p->pack_name, path, path_len); + + strcpy(p->pack_name + path_len, ".keep"); + if (gitfo_exists(p->pack_name) == GIT_SUCCESS) + p->pack_keep = 1; + + strcpy(p->pack_name + path_len, ".pack"); + if (gitfo_stat(p->pack_name, &st) < GIT_SUCCESS || !S_ISREG(st.st_mode)) { + free(p); + return GIT_ENOTFOUND; + } + + /* ok, it looks sane as far as we can check without + * actually mapping the pack file. + */ + p->pack_size = (off_t)st.st_size; + p->pack_local = local; + p->mtime = st.st_mtime; + + /* see if we can parse the sha1 oid in the packfile name */ + if (path_len < 40 || + git_oid_mkstr(&p->sha1, path + path_len - GIT_OID_HEXSZ) < GIT_SUCCESS) + memset(&p->sha1, 0x0, GIT_OID_RAWSZ); + + *pack_out = p; + return GIT_SUCCESS; +} + +static int packfile_load__cb(void *_data, char *path) +{ + struct pack__dirent *data = (struct pack__dirent *)_data; + struct pack_file *pack; + int error; + + if (git__suffixcmp(path, ".idx") != 0) + return GIT_SUCCESS; /* not an index */ + + /* FIXME: git.git checks for duplicate packs. + * But that makes no fucking sense. Our dirent is not + * going to generate dupicate entries */ + + error = packfile_check(&pack, path, data->is_pack_local); + if (error < GIT_SUCCESS) + return error; + + if (git_vector_insert(&data->backend->packs, pack) < GIT_SUCCESS) { + free(pack); + return GIT_ENOMEM; + } + + return GIT_SUCCESS; +} + +static int packfile_load_all(struct pack_backend *backend, const char *odb_path, int local) +{ + int error; + char path[GIT_PATH_MAX]; + struct pack__dirent data; + + data.backend = backend; + data.is_pack_local = local; + + git__joinpath(path, odb_path, "pack"); + if (gitfo_isdir(path) < GIT_SUCCESS) + return GIT_SUCCESS; + + error = gitfo_dirent(path, GIT_PATH_MAX, packfile_load__cb, (void *)&data); + if (error < GIT_SUCCESS) + return error; + + git_vector_sort(&backend->packs); + backend->last_found = git_vector_get(&backend->packs, 0); + + return GIT_SUCCESS; +} + + + + + + + + +/*********************************************************** + * + * PACKFILE ENTRY SEARCH INTERNALS + * + ***********************************************************/ + +static off_t nth_packed_object_offset(const struct pack_file *p, uint32_t n) +{ + const unsigned char *index = p->index_map.data; + index += 4 * 256; + if (p->index_version == 1) { + return ntohl(*((uint32_t *)(index + 24 * n))); + } else { + uint32_t off; + index += 8 + p->num_objects * (20 + 4); + off = ntohl(*((uint32_t *)(index + 4 * n))); + if (!(off & 0x80000000)) + return off; + index += p->num_objects * 4 + (off & 0x7fffffff) * 8; + return (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) | + ntohl(*((uint32_t *)(index + 4))); + } +} + +static int pack_entry_find_offset( + off_t *offset_out, + struct pack_file *p, + const git_oid *oid) +{ + const uint32_t *level1_ofs = p->index_map.data; + const unsigned char *index = p->index_map.data; + unsigned hi, lo, stride; + + *offset_out = 0; + + if (index == NULL) { + int error; + + if ((error = pack_index_open(p)) < GIT_SUCCESS) + return error; + + assert(p->index_map.data); + + index = p->index_map.data; + level1_ofs = p->index_map.data; + } + + if (p->index_version > 1) { + level1_ofs += 2; + index += 8; + } + + index += 4 * 256; + hi = ntohl(level1_ofs[(int)oid->id[0]]); + lo = ((oid->id[0] == 0x0) ? 0 : ntohl(level1_ofs[(int)oid->id[0] - 1])); + + if (p->index_version > 1) { + stride = 20; + } else { + stride = 24; + index += 4; + } + +#ifdef INDEX_DEBUG_LOOKUP + printf("%02x%02x%02x... lo %u hi %u nr %d\n", + oid->id[0], oid->id[1], oid->id[2], lo, hi, p->num_objects); +#endif + +#ifdef GIT2_INDEX_LOOKUP /* TODO: use the advanced lookup method from git.git */ + + int pos = sha1_entry_pos(index, stride, 0, lo, hi, p->num_objects, oid); + if (pos < 0) + return GIT_ENOTFOUND; + + *offset_out = nth_packed_object_offset(p, pos); + return GIT_SUCCESS; + +#else /* use an old and boring binary search */ + + do { + unsigned mi = (lo + hi) / 2; + int cmp = memcmp(index + mi * stride, oid->id, GIT_OID_RAWSZ); + + if (!cmp) { + *offset_out = nth_packed_object_offset(p, mi); + return GIT_SUCCESS; + } + + if (cmp > 0) + hi = mi; + else + lo = mi+1; + + } while (lo < hi); + + return GIT_ENOTFOUND; +#endif +} + +static int pack_entry_find1( + struct pack_entry *e, + struct pack_file *p, + const git_oid *oid) +{ + off_t offset; + + assert(p); + + if (p->num_bad_objects) { + unsigned i; + for (i = 0; i < p->num_bad_objects; i++) + if (git_oid_cmp(oid, &p->bad_object_sha1[i]) == 0) + return GIT_ERROR; + } + + if (pack_entry_find_offset(&offset, p, oid) < GIT_SUCCESS) + return GIT_ENOTFOUND; + + /* we found an entry in the index; + * make sure the packfile backing the index + * still exists on disk */ + if (p->pack_fd == -1 && packfile_open(p) < GIT_SUCCESS) + return GIT_EOSERR; + + e->offset = offset; + e->p = p; + + git_oid_cpy(&e->sha1, oid); + return GIT_SUCCESS; +} + +static int pack_entry_find(struct pack_entry *e, struct pack_backend *backend, const git_oid *oid) +{ + size_t i; + + if (backend->last_found && + pack_entry_find1(e, backend->last_found, oid) == GIT_SUCCESS) + return GIT_SUCCESS; + + for (i = 0; i < backend->packs.length; ++i) { + struct pack_file *p; + + p = git_vector_get(&backend->packs, i); + if (p == backend->last_found) + continue; + + if (pack_entry_find1(e, p, oid) == GIT_SUCCESS) { + backend->last_found = p; + return GIT_SUCCESS; + } + } + + return GIT_ENOTFOUND; +} + + + + + + + + + + + + +/*********************************************************** + * + * PACKFILE ENTRY UNPACK INTERNALS + * + ***********************************************************/ + +static unsigned long packfile_unpack_header1( + size_t *sizep, + git_otype *type, + const unsigned char *buf, + unsigned long len) +{ + unsigned shift; + unsigned long size, c; + unsigned long used = 0; + + c = buf[used++]; + *type = (c >> 4) & 7; + size = c & 15; + shift = 4; + while (c & 0x80) { + if (len <= used || bitsizeof(long) <= shift) + return 0; + + c = buf[used++]; + size += (c & 0x7f) << shift; + shift += 7; + } + + *sizep = (size_t)size; + return used; +} + +static int packfile_unpack_header( + size_t *size_p, + git_otype *type_p, + struct pack_backend *backend, + struct pack_file *p, + struct pack_window **w_curs, + off_t *curpos) +{ + unsigned char *base; + unsigned int left; + unsigned long used; + + /* pack_window_open() assures us we have [base, base + 20) available + * as a range that we can look at at. (Its actually the hash + * size that is assured.) With our object header encoding + * the maximum deflated object size is 2^137, which is just + * insane, so we know won't exceed what we have been given. + */ + base = pack_window_open(backend, p, w_curs, *curpos, &left); + if (base == NULL) + return GIT_ENOMEM; + + used = packfile_unpack_header1(size_p, type_p, base, left); + + if (used == 0) + return GIT_EOBJCORRUPTED; + + *curpos += used; + return GIT_SUCCESS; +} + +static int packfile_unpack_compressed( + git_rawobj *obj, + struct pack_backend *backend, + struct pack_file *p, + struct pack_window **w_curs, + off_t curpos, + size_t size, + git_otype type) +{ + int st; + z_stream stream; + unsigned char *buffer, *in; + + buffer = git__malloc(size); + + memset(&stream, 0, sizeof(stream)); + stream.next_out = buffer; + stream.avail_out = size + 1; + + st = inflateInit(&stream); + if (st != Z_OK) { + free(buffer); + return GIT_EZLIB; + } + + do { + in = pack_window_open(backend, p, w_curs, curpos, &stream.avail_in); + stream.next_in = in; + st = inflate(&stream, Z_FINISH); + + if (!stream.avail_out) + break; /* the payload is larger than it should be */ + + curpos += stream.next_in - in; + } while (st == Z_OK || st == Z_BUF_ERROR); + + inflateEnd(&stream); + + if ((st != Z_STREAM_END) || stream.total_out != size) { + free(buffer); + return GIT_EZLIB; + } + + obj->type = type; + obj->len = size; + obj->data = buffer; + return GIT_SUCCESS; +} + +static off_t get_delta_base( + struct pack_backend *backend, + struct pack_file *p, + struct pack_window **w_curs, + off_t *curpos, + git_otype type, + off_t delta_obj_offset) +{ + unsigned char *base_info = pack_window_open(backend, p, w_curs, *curpos, NULL); + off_t base_offset; + + /* pack_window_open() assured us we have [base_info, base_info + 20) + * as a range that we can look at without walking off the + * end of the mapped window. Its actually the hash size + * that is assured. An OFS_DELTA longer than the hash size + * is stupid, as then a REF_DELTA would be smaller to store. + */ + if (type == GIT_OBJ_OFS_DELTA) { + unsigned used = 0; + unsigned char c = base_info[used++]; + base_offset = c & 127; + while (c & 128) { + base_offset += 1; + if (!base_offset || MSB(base_offset, 7)) + return 0; /* overflow */ + c = base_info[used++]; + base_offset = (base_offset << 7) + (c & 127); + } + base_offset = delta_obj_offset - base_offset; + if (base_offset <= 0 || base_offset >= delta_obj_offset) + return 0; /* out of bound */ + *curpos += used; + } else if (type == GIT_OBJ_REF_DELTA) { + /* The base entry _must_ be in the same pack */ + if (pack_entry_find_offset(&base_offset, p, (git_oid *)base_info) < GIT_SUCCESS) + return GIT_EPACKCORRUPTED; + *curpos += 20; + } else + return 0; + + return base_offset; +} + +static int packfile_unpack_delta( + git_rawobj *obj, + struct pack_backend *backend, + struct pack_file *p, + struct pack_window **w_curs, + off_t curpos, + size_t delta_size, + git_otype delta_type, + off_t obj_offset) +{ + off_t base_offset; + git_rawobj base, delta; + int error; + + base_offset = get_delta_base(backend, p, w_curs, &curpos, delta_type, obj_offset); + if (base_offset == 0) + return GIT_EOBJCORRUPTED; + + pack_window_close(w_curs); + error = packfile_unpack(&base, backend, p, base_offset); + + /* TODO: git.git tries to load the base from other packfiles + * or loose objects */ + if (error < GIT_SUCCESS) + return error; + + error = packfile_unpack_compressed(&delta, backend, p, w_curs, curpos, delta_size, delta_type); + if (error < GIT_SUCCESS) { + git_rawobj_close(&base); + return error; + } + + obj->type = base.type; + error = git__delta_apply(obj, + base.data, base.len, + delta.data, delta.len); + + git_rawobj_close(&base); + git_rawobj_close(&delta); + + /* TODO: we might want to cache this shit. eventually */ + //add_delta_base_cache(p, base_offset, base, base_size, *type); + return error; +} + +static int packfile_unpack( + git_rawobj *obj, + struct pack_backend *backend, + struct pack_file *p, + off_t obj_offset) +{ + struct pack_window *w_curs = NULL; + off_t curpos = obj_offset; + int error; + + size_t size; + git_otype type; + + /* + * TODO: optionally check the CRC on the packfile + */ + + obj->data = NULL; + obj->len = 0; + obj->type = GIT_OBJ_BAD; + + error = packfile_unpack_header(&size, &type, backend, p, &w_curs, &curpos); + if (error < GIT_SUCCESS) + return error; + + switch (type) { + case GIT_OBJ_OFS_DELTA: + case GIT_OBJ_REF_DELTA: + error = packfile_unpack_delta( + obj, backend, p, &w_curs, curpos, + size, type, obj_offset); + break; + + case GIT_OBJ_COMMIT: + case GIT_OBJ_TREE: + case GIT_OBJ_BLOB: + case GIT_OBJ_TAG: + error = packfile_unpack_compressed( + obj, backend, p, &w_curs, curpos, + size, type); + break; + + default: + error = GIT_EOBJCORRUPTED; + break; + } + + pack_window_close(&w_curs); + return error; +} + + + + + /*********************************************************** * * PACKED BACKEND PUBLIC API @@ -1126,79 +1323,80 @@ cleanup: * ***********************************************************/ +/* int pack_backend__read_header(git_rawobj *obj, git_odb_backend *backend, const git_oid *oid) { pack_location location; assert(obj && backend && oid); - if (locate_packfile(&location, (pack_backend *)backend, oid) < 0) + if (locate_packfile(&location, (struct pack_backend *)backend, oid) < 0) return GIT_ENOTFOUND; return read_header_packed(obj, &location); } +*/ int pack_backend__read(git_rawobj *obj, git_odb_backend *backend, const git_oid *oid) { - pack_location location; + struct pack_entry e; + int error; - assert(obj && backend && oid); + if ((error = pack_entry_find(&e, (struct pack_backend *)backend, oid)) < GIT_SUCCESS) + return error; - if (locate_packfile(&location, (pack_backend *)backend, oid) < 0) - return GIT_ENOTFOUND; - - return read_packed(obj, &location); + return packfile_unpack(obj, (struct pack_backend *)backend, e.p, e.offset); } int pack_backend__exists(git_odb_backend *backend, const git_oid *oid) { - pack_location location; - assert(backend && oid); - return locate_packfile(&location, (pack_backend *)backend, oid) == GIT_SUCCESS; + struct pack_entry e; + return pack_entry_find(&e, (struct pack_backend *)backend, oid) == GIT_SUCCESS; } void pack_backend__free(git_odb_backend *_backend) { - pack_backend *backend; - git_packlist *pl; + struct pack_backend *backend; + size_t i; assert(_backend); - backend = (pack_backend *)_backend; + backend = (struct pack_backend *)_backend; - gitlck_lock(&backend->lock); + for (i = 0; i < backend->packs.length; ++i) { + struct pack_file *p = git_vector_get(&backend->packs, i); + packfile_free(backend, p); + } - pl = backend->packlist; - backend->packlist = NULL; - - gitlck_unlock(&backend->lock); - if (pl) - packlist_dec(backend, pl); - - gitlck_free(&backend->lock); - - free(backend->objects_dir); + git_vector_free(&backend->packs); free(backend); } int git_odb_backend_pack(git_odb_backend **backend_out, const char *objects_dir) { - pack_backend *backend; + int error; + struct pack_backend *backend; - backend = git__calloc(1, sizeof(pack_backend)); + backend = git__calloc(1, sizeof(struct pack_backend)); if (backend == NULL) return GIT_ENOMEM; - backend->objects_dir = git__strdup(objects_dir); - if (backend->objects_dir == NULL) { + if (git_vector_init(&backend->packs, 8, packfile_sort__cb) < GIT_SUCCESS) { free(backend); return GIT_ENOMEM; } - gitlck_init(&backend->lock); + backend->window_size = DEFAULT_WINDOW_SIZE; + backend->mapped_limit = DEFAULT_MAPPED_LIMIT; + + error = packfile_load_all(backend, objects_dir, 1); + if (error < GIT_SUCCESS) { + pack_backend__free((git_odb_backend *)backend); + return error; + } backend->parent.read = &pack_backend__read; - backend->parent.read_header = &pack_backend__read_header; + backend->parent.read_header = NULL; backend->parent.write = NULL; backend->parent.exists = &pack_backend__exists; backend->parent.free = &pack_backend__free; diff --git a/src/util.h b/src/util.h index fab93c1c2..e0dfd7b20 100644 --- a/src/util.h +++ b/src/util.h @@ -2,6 +2,8 @@ #define INCLUDE_util_h__ #define ARRAY_SIZE(x) (sizeof(x)/sizeof(x[0])) +#define bitsizeof(x) (CHAR_BIT * sizeof(x)) +#define MSB(x, bits) ((x) & (~0ULL << (bitsizeof(x) - (bits)))) /* * Don't wrap malloc/calloc. From 005718280712634486a097427212e652b0e29f36 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Sat, 12 Mar 2011 16:04:46 +0200 Subject: [PATCH 06/81] Add new method `git_reference_listall` Lists all the references in a repository. Listing may be filtered by reference type. This should applease Lord Clem. --- include/git2/common.h | 16 +++++++++ include/git2/refs.h | 23 ++++++++++++ include/git2/types.h | 1 + src/refs.c | 82 +++++++++++++++++++++++++++++++++++++++++++ tests/t10-refs.c | 13 +++++++ 5 files changed, 135 insertions(+) diff --git a/include/git2/common.h b/include/git2/common.h index 34efe808b..11a08f897 100644 --- a/include/git2/common.h +++ b/include/git2/common.h @@ -27,6 +27,7 @@ #include "thread-utils.h" #include +#include #ifdef __cplusplus # define GIT_BEGIN_DECL extern "C" { @@ -158,6 +159,21 @@ #define GIT_EINVALIDREFSTATE (GIT_ERROR - 21) GIT_BEGIN_DECL + +typedef struct { + char **strings; + size_t count; +} git_strarray; + +GIT_INLINE(void) git_strarray_free(git_strarray *array) +{ + size_t i; + for (i = 0; i < array->count; ++i) + free(array->strings[i]); + + free(array->strings); +} + /** @} */ GIT_END_DECL #endif diff --git a/include/git2/refs.h b/include/git2/refs.h index 1702d7ee1..4ffc5ce5b 100644 --- a/include/git2/refs.h +++ b/include/git2/refs.h @@ -218,6 +218,29 @@ GIT_EXTERN(int) git_reference_delete(git_reference *ref); */ GIT_EXTERN(int) git_reference_packall(git_repository *repo); +/** + * Fill a list with all the references that can be found + * in a repository. + * + * The listed references may be filtered by type, or using + * a bitwise OR of several types. Use the magic value + * `GIT_REF_LISTALL` to obtain all references, including + * packed ones. + * + * The string array will be filled with the names of all + * references; these values are owned by the user and + * should be free'd manually when no longer needed, using + * `git_strarray_free`. + * + * @param array Pointer to a git_strarray structure where + * the reference names will be stored + * @param repo Repository where to find the refs + * @param list_flags Filtering flags for the reference + * listing. + * @return 0 on success; error code otherwise + */ +GIT_EXTERN(int) git_reference_listall(git_strarray *array, git_repository *repo, unsigned int list_flags); + /** @} */ GIT_END_DECL #endif diff --git a/include/git2/types.h b/include/git2/types.h index 62467ec45..b5a8d7b2d 100644 --- a/include/git2/types.h +++ b/include/git2/types.h @@ -145,6 +145,7 @@ typedef enum { GIT_REF_SYMBOLIC = 2, /** A reference which points at another reference */ GIT_REF_PACKED = 4, GIT_REF_HAS_PEEL = 8, + GIT_REF_LISTALL = GIT_REF_OID|GIT_REF_SYMBOLIC|GIT_REF_PACKED, } git_rtype; /** @} */ diff --git a/src/refs.c b/src/refs.c index 2fc383e22..93897a7f9 100644 --- a/src/refs.c +++ b/src/refs.c @@ -235,6 +235,24 @@ static int loose_read(gitfo_buf *file_content, const char *name, const char *rep return error; } +static git_rtype loose_guess_rtype(const char *full_path) +{ + gitfo_buf ref_file = GITFO_BUF_INIT; + git_rtype type; + + type = GIT_REF_INVALID; + + if (gitfo_read_file(&ref_file, full_path) == GIT_SUCCESS) { + if (git__prefixcmp((const char *)(ref_file.data), GIT_SYMREF) == 0) + type = GIT_REF_SYMBOLIC; + else + type = GIT_REF_OID; + } + + gitfo_free_buf(&ref_file); + return type; +} + static int loose_lookup( git_reference **ref_out, git_repository *repo, @@ -531,6 +549,31 @@ cleanup: return error; } + + + +struct dirent_list_data { + git_vector ref_list; + size_t repo_path_len; + unsigned int list_flags; +}; + +static int _dirent_loose_listall(void *_data, char *full_path) +{ + struct dirent_list_data *data = (struct dirent_list_data *)_data; + char *file_path; + + if (gitfo_isdir(full_path) == GIT_SUCCESS) + return gitfo_dirent(full_path, GIT_PATH_MAX, _dirent_loose_listall, _data); + + if ((data->list_flags & loose_guess_rtype(full_path)) == 0) + return GIT_SUCCESS; /* we are filtering out this reference */ + + file_path = full_path + data->repo_path_len; + + return git_vector_insert(&data->ref_list, git__strdup(file_path)); +} + static int _dirent_loose_load(void *data, char *full_path) { git_repository *repository = (git_repository *)data; @@ -1292,6 +1335,45 @@ int git_reference_packall(git_repository *repo) return packed_write(repo); } +int git_reference_listall(git_strarray *array, git_repository *repo, unsigned int list_flags) +{ + int error; + struct dirent_list_data data; + char refs_path[GIT_PATH_MAX]; + + array->strings = NULL; + array->count = 0; + + git_vector_init(&data.ref_list, 8, NULL); + data.repo_path_len = strlen(repo->path_repository); + data.list_flags = list_flags; + + git__joinpath(refs_path, repo->path_repository, GIT_REFS_DIR); + error = gitfo_dirent(refs_path, GIT_PATH_MAX, _dirent_loose_listall, &data); + + if (error < GIT_SUCCESS) { + git_vector_free(&data.ref_list); + return error; + } + + if (list_flags & GIT_REF_PACKED) { + const char *ref_name; + void *_unused; + + if ((error = packed_load(repo)) < GIT_SUCCESS) { + git_vector_free(&data.ref_list); + return error; + } + + GIT_HASHTABLE_FOREACH(repo->references.packfile, ref_name, _unused, + git_vector_insert(&data.ref_list, git__strdup(ref_name)); + ); + } + + array->strings = (char **)data.ref_list.contents; + array->count = data.ref_list.length; + return GIT_SUCCESS; +} diff --git a/tests/t10-refs.c b/tests/t10-refs.c index abe364133..c70fb69ce 100644 --- a/tests/t10-refs.c +++ b/tests/t10-refs.c @@ -710,6 +710,18 @@ BEGIN_TEST(normalize2, "tests borrowed from JGit") must_fail(ensure_refname_normalized(SYM_REF, "refs/heads/master@{1.hour.ago}", NULL)); END_TEST +BEGIN_TEST(list0, "try to list all the references in our test repo") + git_repository *repo; + git_strarray ref_list; + + must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); + must_pass(git_reference_listall(&ref_list, repo, GIT_REF_LISTALL)); + must_be_true(ref_list.count == 8); /* 8 refs in total if we include the packed ones */ + + git_strarray_free(&ref_list); + git_repository_free(repo); +END_TEST + BEGIN_SUITE(refs) ADD_TEST(readtag0); @@ -741,4 +753,5 @@ BEGIN_SUITE(refs) ADD_TEST(rename4); ADD_TEST(delete0); + ADD_TEST(list0); END_SUITE From 6b2a19418cb08e7bccefc2362f290a67555594a0 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Sat, 12 Mar 2011 23:09:16 +0200 Subject: [PATCH 07/81] Fix the retarded object interdependency system It's no longer retarded. All object interdependencies are stored as OIDs instead of actual objects. This should be hundreds of times faster, specially on big repositories. Heck, who knows, maye it doesn't even segfault -- wouldn't that be awesome? What has changed on the API? `git_commit_parent`, `git_commit_tree`, `git_tag_target` now return their values through a pointer-to-pointer, and have an error code. `git_commit_set_tree` and `git_tag_set_target` now return an error code and may fail. `git_repository_free__no_gc` has been deprecated because it's stupid. Since there are no longer any interdependencies between objects, we don't need internal reference counting, and GC never fails or double-free's pointers. `git_object_close` now does a very sane thing: marks an object as unused. Closed objects will be eventually free'd from the object cache based on LRU. Please use `git_object_close` from the garbage collector `destroy` method on your bindings. It's 100% safe. `git_repository_gc` is a new method that forces a garbage collector pass through the repo, to free as many LRU objects as possible. This is useful if we are running out of memory. --- include/git2/commit.h | 16 +-- include/git2/object.h | 8 +- include/git2/repository.h | 12 ++- include/git2/tag.h | 15 ++- src/blob.c | 4 +- src/commit.c | 212 ++++++++++++++------------------------ src/commit.h | 7 +- src/object.c | 22 ++-- src/refs.c | 7 +- src/repository.c | 83 ++++++--------- src/repository.h | 22 +--- src/tag.c | 36 +++---- src/tag.h | 3 +- tests/t04-commit.c | 4 +- tests/t08-tag.c | 4 +- 15 files changed, 187 insertions(+), 268 deletions(-) diff --git a/include/git2/commit.h b/include/git2/commit.h index 21836dbbd..1556e52b1 100644 --- a/include/git2/commit.h +++ b/include/git2/commit.h @@ -122,10 +122,11 @@ GIT_EXTERN(const git_signature *) git_commit_author(git_commit *commit); /** * Get the tree pointed to by a commit. + * @param tree_out pointer where to store the tree object * @param commit a previously loaded commit. - * @return the tree of a commit + * @return 0 on success; error code otherwise */ -GIT_EXTERN(const git_tree *) git_commit_tree(git_commit *commit); +GIT_EXTERN(int) git_commit_tree(git_tree **tree_out, git_commit *commit); /** * Get the number of parents of this commit @@ -137,11 +138,13 @@ GIT_EXTERN(unsigned int) git_commit_parentcount(git_commit *commit); /** * Get the specified parent of the commit. + * + * @param parent Pointer where to store the parent commit * @param commit a previously loaded commit. - * @param n the position of the entry - * @return a pointer to the commit; NULL if out of bounds + * @param n the position of the parent (from 0 to `parentcount`) + * @return 0 on success; error code otherwise */ -GIT_EXTERN(git_commit *) git_commit_parent(git_commit *commit, unsigned int n); +GIT_EXTERN(int) git_commit_parent(git_commit **parent, git_commit *commit, unsigned int n); /** * Add a new parent commit to an existing commit @@ -176,8 +179,9 @@ GIT_EXTERN(void) git_commit_set_author(git_commit *commit, const git_signature * * Set the tree which is pointed to by a commit * @param commit the commit object * @param tree the new tree + * @param 0 on success; error code otherwise */ -GIT_EXTERN(void) git_commit_set_tree(git_commit *commit, git_tree *tree); +GIT_EXTERN(int) git_commit_set_tree(git_commit *commit, git_tree *tree); /** @} */ GIT_END_DECL diff --git a/include/git2/object.h b/include/git2/object.h index af0f014e3..748386f69 100644 --- a/include/git2/object.h +++ b/include/git2/object.h @@ -131,10 +131,10 @@ GIT_EXTERN(git_repository *) git_object_owner(const git_object *obj); * Close an open object * * This method instructs the library to close an existing - * object; note that git_objects are owned by the repository - * and are reference counted, so the object may or may not be - * freed after this library call, depending on whether any other - * objects still depend on it. + * object; note that git_objects are owned and cached by the repository + * so the object may or may not be freed after this library call, + * depending on how agressive is the caching mechanism used + * by the repository. * * IMPORTANT: * It is *not* necessary to call this method when you stop using diff --git a/include/git2/repository.h b/include/git2/repository.h index 5327f8c58..d5f6cf501 100644 --- a/include/git2/repository.h +++ b/include/git2/repository.h @@ -154,12 +154,20 @@ GIT_EXTERN(int) git_repository_index(git_index **index, git_repository *repo); /** * Free a previously allocated repository + * * @param repo repository handle to close. If NULL nothing occurs. */ GIT_EXTERN(void) git_repository_free(git_repository *repo); - -GIT_EXTERN(void) git_repository_free__no_gc(git_repository *repo); +/** + * Force a garbage collector pass on the repository + * + * This will force-free any cached objects that have been + * previously marked by the user as closed (`git_object_close`). + * + * @param repo repository handle to collect. If NULL nothing occurs. + */ +GIT_EXTERN(void) git_repository_close(git_repository *repo); /** * Creates a new Git repository in the given folder. diff --git a/include/git2/tag.h b/include/git2/tag.h index 2ca25c8a0..f1669eb90 100644 --- a/include/git2/tag.h +++ b/include/git2/tag.h @@ -79,10 +79,18 @@ GIT_EXTERN(const git_oid *) git_tag_id(git_tag *tag); /** * Get the tagged object of a tag + * @param target pointer where to store the target * @param tag a previously loaded tag. - * @return reference to a repository object + * @return 0 on success; error code otherwise */ -GIT_EXTERN(const git_object *) git_tag_target(git_tag *t); +GIT_EXTERN(int) git_tag_target(git_object **target, git_tag *t); + +/** + * Get the OID of the tagged object of a tag + * @param tag a previously loaded tag. + * @return pointer to the OID + */ +GIT_EXTERN(const git_oid *) git_tag_target_oid(git_tag *t); /** * Get the type of a tag's tagged object @@ -117,7 +125,7 @@ GIT_EXTERN(const char *) git_tag_message(git_tag *t); * @param tag The tag to modify * @param target the new tagged target */ -GIT_EXTERN(void) git_tag_set_target(git_tag *tag, git_object *target); +GIT_EXTERN(int) git_tag_set_target(git_tag *tag, git_object *target); /** * Set the name of a tag @@ -130,6 +138,7 @@ GIT_EXTERN(void) git_tag_set_name(git_tag *tag, const char *name); * Set the tagger of a tag * @param tag The tag to modify * @param tagger_sig signature of the tagging action + * @return 0 on success; error code otherwise */ GIT_EXTERN(void) git_tag_set_tagger(git_tag *tag, const git_signature *tagger_sig); diff --git a/src/blob.c b/src/blob.c index f157f4787..1e03b6b67 100644 --- a/src/blob.c +++ b/src/blob.c @@ -130,9 +130,7 @@ int git_blob_writefile(git_oid *written_id, git_repository *repo, const char *pa git_oid_cpy(written_id, git_object_id((git_object *)blob)); - /* FIXME: maybe we don't want to free this already? - * the user may want to access it again */ - GIT_OBJECT_DECREF(repo, blob); + git_object_close((git_object*)blob); return GIT_SUCCESS; } diff --git a/src/commit.c b/src/commit.c index 974999a4a..1c5cddf7a 100644 --- a/src/commit.c +++ b/src/commit.c @@ -46,24 +46,22 @@ static void clear_parents(git_commit *commit) { unsigned int i; - for (i = 0; i < commit->parents.length; ++i) { - git_commit *parent = git_vector_get(&commit->parents, i); - GIT_OBJECT_DECREF(commit->object.repo, parent); + for (i = 0; i < commit->parent_oids.length; ++i) { + git_oid *parent = git_vector_get(&commit->parent_oids, i); + free(parent); } - git_vector_clear(&commit->parents); + git_vector_clear(&commit->parent_oids); } void git_commit__free(git_commit *commit) { clear_parents(commit); - git_vector_free(&commit->parents); + git_vector_free(&commit->parent_oids); git_signature_free(commit->author); git_signature_free(commit->committer); - GIT_OBJECT_DECREF(commit->object.repo, commit->tree); - free(commit->message); free(commit->message_short); free(commit); @@ -78,16 +76,13 @@ int git_commit__writeback(git_commit *commit, git_odb_source *src) { unsigned int i; - if (commit->tree == NULL) - return GIT_EMISSINGOBJDATA; + git__write_oid(src, "tree", &commit->tree_oid); - git__write_oid(src, "tree", git_tree_id(commit->tree)); + for (i = 0; i < commit->parent_oids.length; ++i) { + git_oid *parent_oid; - for (i = 0; i < commit->parents.length; ++i) { - git_commit *parent; - - parent = git_vector_get(&commit->parents, i); - git__write_oid(src, "parent", git_commit_id(parent)); + parent_oid = git_vector_get(&commit->parent_oids, i); + git__write_oid(src, "parent", parent_oid); } if (commit->author == NULL) @@ -103,66 +98,49 @@ int git_commit__writeback(git_commit *commit, git_odb_source *src) if (commit->message != NULL) { git__source_write(src, "\n", 1); git__source_write(src, commit->message, strlen(commit->message)); - } - - /* Mark the commit as having all attributes */ - commit->full_parse = 1; + } return GIT_SUCCESS; } -int commit_parse_buffer(git_commit *commit, void *data, size_t len, unsigned int parse_flags) +int commit_parse_buffer(git_commit *commit, void *data, size_t len) { char *buffer = (char *)data; const char *buffer_end = (char *)data + len; - git_oid oid; + git_oid parent_oid; int error; /* first parse; the vector hasn't been initialized yet */ - if (commit->parents.contents == NULL) { - git_vector_init(&commit->parents, 4, NULL); + if (commit->parent_oids.contents == NULL) { + git_vector_init(&commit->parent_oids, 4, NULL); } clear_parents(commit); - - if ((error = git__parse_oid(&oid, &buffer, buffer_end, "tree ")) < GIT_SUCCESS) - return error; - - GIT_OBJECT_DECREF(commit->object.repo, commit->tree); - if ((error = git_object_lookup((git_object **)&commit->tree, commit->object.repo, &oid, GIT_OBJ_TREE)) < GIT_SUCCESS) + if ((error = git__parse_oid(&commit->tree_oid, &buffer, buffer_end, "tree ")) < GIT_SUCCESS) return error; /* * TODO: commit grafts! */ - while (git__parse_oid(&oid, &buffer, buffer_end, "parent ") == GIT_SUCCESS) { - git_commit *parent; + while (git__parse_oid(&parent_oid, &buffer, buffer_end, "parent ") == GIT_SUCCESS) { + git_oid *new_oid; - if ((error = git_object_lookup((git_object **)&parent, commit->object.repo, &oid, GIT_OBJ_COMMIT)) < GIT_SUCCESS) - return error; + new_oid = git__malloc(sizeof(git_oid)); + git_oid_cpy(new_oid, &parent_oid); - if (git_vector_insert(&commit->parents, parent) < GIT_SUCCESS) + if (git_vector_insert(&commit->parent_oids, new_oid) < GIT_SUCCESS) return GIT_ENOMEM; } + if (commit->author) + git_signature_free(commit->author); - if (parse_flags & COMMIT_FULL_PARSE) { - if (commit->author) - git_signature_free(commit->author); - - commit->author = git__malloc(sizeof(git_signature)); - if ((error = git_signature__parse(commit->author, &buffer, buffer_end, "author ")) < GIT_SUCCESS) - return error; - - } else { - if ((buffer = memchr(buffer, '\n', buffer_end - buffer)) == NULL) - return GIT_EOBJCORRUPTED; - - buffer++; - } + commit->author = git__malloc(sizeof(git_signature)); + if ((error = git_signature__parse(commit->author, &buffer, buffer_end, "author ")) < GIT_SUCCESS) + return error; /* Always parse the committer; we need the commit time */ if (commit->committer) @@ -176,7 +154,7 @@ int commit_parse_buffer(git_commit *commit, void *data, size_t len, unsigned int while (buffer <= buffer_end && *buffer == '\n') buffer++; - if (parse_flags & COMMIT_FULL_PARSE && buffer < buffer_end) { + if (buffer < buffer_end) { const char *line_end; size_t message_len = buffer_end - buffer; @@ -203,106 +181,81 @@ int git_commit__parse(git_commit *commit) { assert(commit && commit->object.source.open); return commit_parse_buffer(commit, - commit->object.source.raw.data, commit->object.source.raw.len, COMMIT_BASIC_PARSE); + commit->object.source.raw.data, commit->object.source.raw.len); } -int git_commit__parse_full(git_commit *commit) -{ - int error; - - if (commit->full_parse) - return GIT_SUCCESS; - - if ((error = git_object__source_open((git_object *)commit)) < GIT_SUCCESS) - return error; - - error = commit_parse_buffer(commit, - commit->object.source.raw.data, commit->object.source.raw.len, COMMIT_FULL_PARSE); - - git_object__source_close((git_object *)commit); - - commit->full_parse = 1; - return error; -} - - - -#define GIT_COMMIT_GETTER(_rvalue, _name) \ - const _rvalue git_commit_##_name(git_commit *commit) \ +#define GIT_COMMIT_GETTER(_rvalue, _name, _return) \ + _rvalue git_commit_##_name(git_commit *commit) \ {\ assert(commit); \ - if (commit->_name) \ - return commit->_name; \ - if (!commit->object.in_memory) \ - git_commit__parse_full(commit); \ - return commit->_name; \ + return _return; \ } -#define CHECK_FULL_PARSE() \ - if (!commit->object.in_memory && !commit->full_parse)\ - git_commit__parse_full(commit); +GIT_COMMIT_GETTER(const git_signature *, author, commit->author) +GIT_COMMIT_GETTER(const git_signature *, committer, commit->committer) +GIT_COMMIT_GETTER(const char *, message, commit->message) +GIT_COMMIT_GETTER(const char *, message_short, commit->message_short) +GIT_COMMIT_GETTER(time_t, time, commit->committer->when.time) +GIT_COMMIT_GETTER(int, time_offset, commit->committer->when.offset) +GIT_COMMIT_GETTER(unsigned int, parentcount, commit->parent_oids.length) -const git_tree *git_commit_tree(git_commit *commit) + +int git_commit_tree(git_tree **tree_out, git_commit *commit) { assert(commit); - - if (!commit->object.in_memory && commit->tree == NULL) - git_commit__parse_full(commit); - - GIT_OBJECT_INCREF(commit->object.repo, commit->tree); - return commit->tree; + return git_tree_lookup(tree_out, commit->object.repo, &commit->tree_oid); } -GIT_COMMIT_GETTER(git_signature *, author) -GIT_COMMIT_GETTER(git_signature *, committer) -GIT_COMMIT_GETTER(char *, message) -GIT_COMMIT_GETTER(char *, message_short) - -time_t git_commit_time(git_commit *commit) +int git_commit_parent(git_commit **parent, git_commit *commit, unsigned int n) { - assert(commit && commit->committer); - return commit->committer->when.time; -} - -int git_commit_time_offset(git_commit *commit) -{ - assert(commit && commit->committer); - return commit->committer->when.offset; -} - -unsigned int git_commit_parentcount(git_commit *commit) -{ - assert(commit); - return commit->parents.length; -} - -git_commit *git_commit_parent(git_commit *commit, unsigned int n) -{ - git_commit *parent; - + git_oid *parent_oid; assert(commit); - parent = git_vector_get(&commit->parents, n); - GIT_OBJECT_INCREF(commit->object.repo, parent); - return parent; + parent_oid = git_vector_get(&commit->parent_oids, n); + if (parent_oid == NULL) + return GIT_ENOTFOUND; + + return git_commit_lookup(parent, commit->object.repo, parent_oid); } -void git_commit_set_tree(git_commit *commit, git_tree *tree) + + +int git_commit_set_tree(git_commit *commit, git_tree *tree) { + const git_oid *oid; + assert(commit && tree); - commit->object.modified = 1; - CHECK_FULL_PARSE(); - GIT_OBJECT_DECREF(commit->object.repo, commit->tree); - GIT_OBJECT_INCREF(commit->object.repo, tree); - commit->tree = tree; + if ((oid = git_object_id((git_object *)tree)) == NULL) + return GIT_EMISSINGOBJDATA; + + commit->object.modified = 1; + git_oid_cpy(&commit->tree_oid, oid); + return GIT_SUCCESS; +} + +int git_commit_add_parent(git_commit *commit, git_commit *new_parent) +{ + const git_oid *parent_oid; + git_oid *new_oid; + assert(commit && new_parent); + + if ((parent_oid = git_object_id((git_object *)new_parent)) == NULL) + return GIT_EMISSINGOBJDATA; + + new_oid = git__malloc(sizeof(git_oid)); + if (new_oid == NULL) + return GIT_ENOMEM; + + commit->object.modified = 1; + git_oid_cpy(new_oid, parent_oid); + return git_vector_insert(&commit->parent_oids, new_oid); } void git_commit_set_author(git_commit *commit, const git_signature *author_sig) { assert(commit && author_sig); commit->object.modified = 1; - CHECK_FULL_PARSE(); git_signature_free(commit->author); commit->author = git_signature_dup(author_sig); @@ -312,7 +265,6 @@ void git_commit_set_committer(git_commit *commit, const git_signature *committer { assert(commit && committer_sig); commit->object.modified = 1; - CHECK_FULL_PARSE(); git_signature_free(commit->committer); commit->committer = git_signature_dup(committer_sig); @@ -324,7 +276,6 @@ void git_commit_set_message(git_commit *commit, const char *message) size_t message_len; commit->object.modified = 1; - CHECK_FULL_PARSE(); if (commit->message) free(commit->message); @@ -347,12 +298,3 @@ void git_commit_set_message(git_commit *commit, const char *message) commit->message_short[message_len] = 0; } -int git_commit_add_parent(git_commit *commit, git_commit *new_parent) -{ - assert(commit && new_parent); - - CHECK_FULL_PARSE(); - commit->object.modified = 1; - GIT_OBJECT_INCREF(commit->object.repo, new_parent); - return git_vector_insert(&commit->parents, new_parent); -} diff --git a/src/commit.h b/src/commit.h index b53ee9b23..aaf349ca6 100644 --- a/src/commit.h +++ b/src/commit.h @@ -11,21 +11,18 @@ struct git_commit { git_object object; - git_vector parents; + git_vector parent_oids; + git_oid tree_oid; - git_tree *tree; git_signature *author; git_signature *committer; char *message; char *message_short; - - unsigned full_parse:1; }; void git_commit__free(git_commit *c); int git_commit__parse(git_commit *commit); -int git_commit__parse_full(git_commit *commit); int git_commit__writeback(git_commit *commit, git_odb_source *src); diff --git a/src/object.c b/src/object.c index 7891893a0..fce99153b 100644 --- a/src/object.c +++ b/src/object.c @@ -261,7 +261,7 @@ int git_object_new(git_object **object_out, git_repository *repo, git_otype type object->source.raw.type = type; - object->refcount++; + object->lru = ++repo->lru_counter; *object_out = object; return GIT_SUCCESS; } @@ -277,7 +277,8 @@ int git_object_lookup(git_object **object_out, git_repository *repo, const git_o object = git_hashtable_lookup(repo->objects, id); if (object != NULL) { *object_out = object; - GIT_OBJECT_INCREF(repo, object); + object->lru = ++repo->lru_counter; + object->can_free = 0; return GIT_SUCCESS; } @@ -330,7 +331,7 @@ int git_object_lookup(git_object **object_out, git_repository *repo, const git_o git_object__source_close(object); git_hashtable_insert(repo->objects, &object->id, object); - GIT_OBJECT_INCREF(repo, object); + object->lru = ++repo->lru_counter; *object_out = object; return GIT_SUCCESS; } @@ -412,17 +413,12 @@ void git_object_close(git_object *object) if (object == NULL) return; - if (--object->refcount <= 0) { - if (object->repo != NULL) { - if (object->in_memory) { - int idx = git_vector_search(&object->repo->memory_objects, object); - git_vector_remove(&object->repo->memory_objects, idx); - } else { - git_hashtable_remove(object->repo->objects, &object->id); - } - } - + if (object->in_memory) { + int idx = git_vector_search(&object->repo->memory_objects, object); + git_vector_remove(&object->repo->memory_objects, idx); git_object__free(object); + } else { + object->can_free = 1; } } diff --git a/src/refs.c b/src/refs.c index 93897a7f9..1ed6b567c 100644 --- a/src/refs.c +++ b/src/refs.c @@ -681,7 +681,6 @@ static int packed_write_ref(reference_oid *ref, git_filebuf *file) static int packed_find_peel(reference_oid *ref) { git_tag *tag; - const git_object *peeled_target; int error; if (ref->ref.type & GIT_REF_HAS_PEEL) @@ -706,11 +705,7 @@ static int packed_find_peel(reference_oid *ref) /* * Find the object pointed at by this tag */ - peeled_target = git_tag_target(tag); - if (peeled_target == NULL) - return GIT_EOBJCORRUPTED; - - git_oid_cpy(&ref->peel_target, git_object_id(peeled_target)); + git_oid_cpy(&ref->peel_target, git_tag_target_oid(tag)); ref->ref.type |= GIT_REF_HAS_PEEL; /* diff --git a/src/repository.c b/src/repository.c index c24560665..37aa44781 100644 --- a/src/repository.c +++ b/src/repository.c @@ -209,7 +209,6 @@ static git_repository *repository_alloc() return NULL; } - repo->gc_enabled = 1; return repo; } @@ -331,9 +330,40 @@ cleanup: return error; } -static void repository_free(git_repository *repo) +int git_repository_gc(git_repository *repo) { - assert(repo); + int collected = 0; + git_object *object; + const void *_unused; + + GIT_HASHTABLE_FOREACH(repo->objects, _unused, object, + if (object->can_free) { + git_object__free(object); + collected++; + } + ); + + return collected; +} + +void git_repository_free(git_repository *repo) +{ + git_object *object; + const void *_unused; + unsigned int i; + + if (repo == NULL) + return; + + /* force free all the objects */ + GIT_HASHTABLE_FOREACH(repo->objects, _unused, object, + git_object__free(object); + ); + + for (i = 0; i < repo->memory_objects.length; ++i) { + object = git_vector_get(&repo->memory_objects, i); + git_object__free(object); + } free(repo->path_workdir); free(repo->path_index); @@ -354,53 +384,6 @@ static void repository_free(git_repository *repo) free(repo); } -void git_repository_free__no_gc(git_repository *repo) -{ - git_object *object; - const void *_unused; - unsigned int i; - - if (repo == NULL) - return; - - GIT_HASHTABLE_FOREACH(repo->objects, _unused, object, - object->repo = NULL; - object->refcount = 0; - ); - - for (i = 0; i < repo->memory_objects.length; ++i) { - object = git_vector_get(&repo->memory_objects, i); - object->repo = NULL; - object->refcount = 0; - } - - repository_free(repo); -} - -void git_repository_free(git_repository *repo) -{ - git_object *object; - const void *_unused; - unsigned int i; - - if (repo == NULL) - return; - - repo->gc_enabled = 0; - - /* force free all the objects */ - GIT_HASHTABLE_FOREACH(repo->objects, _unused, object, - git_object__free(object); - ); - - for (i = 0; i < repo->memory_objects.length; ++i) { - object = git_vector_get(&repo->memory_objects, i); - git_object__free(object); - } - - repository_free(repo); -} - int git_repository_index(git_index **index_out, git_repository *repo) { int error; diff --git a/src/repository.h b/src/repository.h index 5318ed45c..48b8dae6b 100644 --- a/src/repository.h +++ b/src/repository.h @@ -27,8 +27,8 @@ struct git_object { git_oid id; git_repository *repo; git_odb_source source; - unsigned short refcount; - unsigned char in_memory, modified; + unsigned int lru; + unsigned char in_memory, modified, can_free, _pad; }; struct git_repository { @@ -45,7 +45,8 @@ struct git_repository { char *path_odb; char *path_workdir; - unsigned is_bare:1, gc_enabled:1; + unsigned is_bare:1; + unsigned int lru_counter; }; int git_object__source_open(git_object *object); @@ -61,19 +62,4 @@ int git__source_write(git_odb_source *source, const void *bytes, size_t len); int git__parse_oid(git_oid *oid, char **buffer_out, const char *buffer_end, const char *header); int git__write_oid(git_odb_source *src, const char *header, const git_oid *oid); -#define GIT_OBJECT_INCREF(repo, ob) git_object__incref((repo), (git_object *)(ob)) -#define GIT_OBJECT_DECREF(repo, ob) git_object__decref((repo), (git_object *)(ob)) - -GIT_INLINE(void) git_object__incref(git_repository *repo, struct git_object *object) -{ - if (repo && repo->gc_enabled && object) - object->refcount++; -} - -GIT_INLINE(void) git_object__decref(git_repository *repo, struct git_object *object) -{ - if (repo && repo->gc_enabled && object) - git_object_close(object); -} - #endif diff --git a/src/tag.c b/src/tag.c index 1ac2067c2..1379425a1 100644 --- a/src/tag.c +++ b/src/tag.c @@ -35,7 +35,6 @@ void git_tag__free(git_tag *tag) { git_signature_free(tag->tagger); - GIT_OBJECT_DECREF(tag->object.repo, tag->target); free(tag->message); free(tag->tag_name); free(tag); @@ -46,23 +45,31 @@ const git_oid *git_tag_id(git_tag *c) return git_object_id((git_object *)c); } -const git_object *git_tag_target(git_tag *t) +int git_tag_target(git_object **target, git_tag *t) { assert(t); - GIT_OBJECT_INCREF(t->object.repo, t->target); - return t->target; + return git_object_lookup(target, t->object.repo, &t->target, t->type); } -void git_tag_set_target(git_tag *tag, git_object *target) +const git_oid *git_tag_target_oid(git_tag *t) { + assert(t); + return &t->target; +} + +int git_tag_set_target(git_tag *tag, git_object *target) +{ + const git_oid *oid; + assert(tag && target); - GIT_OBJECT_DECREF(tag->object.repo, tag->target); - GIT_OBJECT_INCREF(tag->object.repo, target); + if ((oid = git_object_id(target)) == NULL) + return GIT_EMISSINGOBJDATA; tag->object.modified = 1; - tag->target = target; + git_oid_cpy(&tag->target, oid); tag->type = git_object_type(target); + return GIT_SUCCESS; } git_otype git_tag_type(git_tag *t) @@ -81,7 +88,6 @@ void git_tag_set_name(git_tag *tag, const char *name) { assert(tag && name); - /* TODO: sanity check? no newlines in message */ tag->object.modified = 1; if (tag->tag_name) @@ -128,12 +134,11 @@ static int parse_tag_buffer(git_tag *tag, char *buffer, const char *buffer_end) NULL, "commit\n", "tree\n", "blob\n", "tag\n" }; - git_oid target_oid; unsigned int i, text_len; char *search; int error; - if ((error = git__parse_oid(&target_oid, &buffer, buffer_end, "object ")) < 0) + if ((error = git__parse_oid(&tag->target, &buffer, buffer_end, "object ")) < 0) return error; if (buffer + 5 >= buffer_end) @@ -161,11 +166,6 @@ static int parse_tag_buffer(git_tag *tag, char *buffer, const char *buffer_end) if (tag->type == GIT_OBJ_BAD) return GIT_EOBJCORRUPTED; - git_object_close(tag->target); - error = git_object_lookup(&tag->target, tag->object.repo, &target_oid, tag->type); - if (error < 0) - return error; - if (buffer + 4 >= buffer_end) return GIT_EOBJCORRUPTED; @@ -210,10 +210,10 @@ static int parse_tag_buffer(git_tag *tag, char *buffer, const char *buffer_end) int git_tag__writeback(git_tag *tag, git_odb_source *src) { - if (tag->target == NULL || tag->tag_name == NULL || tag->tagger == NULL) + if (tag->tag_name == NULL || tag->tagger == NULL) return GIT_EMISSINGOBJDATA; - git__write_oid(src, "object", git_object_id(tag->target)); + git__write_oid(src, "object", &tag->target); git__source_printf(src, "type %s\n", git_object_type2string(tag->type)); git__source_printf(src, "tag %s\n", tag->tag_name); git_signature__write(src, "tagger", tag->tagger); diff --git a/src/tag.h b/src/tag.h index 624fcc654..a1782d064 100644 --- a/src/tag.h +++ b/src/tag.h @@ -7,8 +7,9 @@ struct git_tag { git_object object; - git_object *target; + git_oid target; git_otype type; + char *tag_name; git_signature *tagger; char *message; diff --git a/tests/t04-commit.c b/tests/t04-commit.c index 8e62759a8..855cf9859 100644 --- a/tests/t04-commit.c +++ b/tests/t04-commit.c @@ -390,11 +390,11 @@ BEGIN_TEST(details0, "query the details on a parsed commit") must_be_true(commit_time > 0); must_be_true(parents <= 2); for (p = 0;p < parents;p++) { - parent = git_commit_parent(commit, p); + must_pass(git_commit_parent(&parent, commit, p)); must_be_true(parent != NULL); must_be_true(git_commit_author(parent) != NULL); // is it really a commit? } - must_be_true(git_commit_parent(commit, parents) == NULL); + must_fail(git_commit_parent(&parent, commit, parents)); } git_repository_free(repo); diff --git a/tests/t08-tag.c b/tests/t08-tag.c index 0afdf719d..c6789266c 100644 --- a/tests/t08-tag.c +++ b/tests/t08-tag.c @@ -48,12 +48,12 @@ BEGIN_TEST(read0, "read and parse a tag from the repository") must_be_true(strcmp(git_tag_name(tag1), "test") == 0); must_be_true(git_tag_type(tag1) == GIT_OBJ_TAG); - tag2 = (git_tag *)git_tag_target(tag1); + must_pass(git_tag_target((git_object **)&tag2, tag1)); must_be_true(tag2 != NULL); must_be_true(git_oid_cmp(&id2, git_tag_id(tag2)) == 0); - commit = (git_commit *)git_tag_target(tag2); + must_pass(git_tag_target((git_object **)&commit, tag2)); must_be_true(commit != NULL); must_be_true(git_oid_cmp(&id_commit, git_commit_id(commit)) == 0); From 7064938bd5e7ef47bfd79a685a62c1e2649e2ce7 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Mon, 14 Mar 2011 23:55:32 +0200 Subject: [PATCH 08/81] libgit2 version 0.10.0, "very disco" A version *so* awesome that needs 2 version bumps AND a codename. Major features: - New internal garbage collection (harder) - Pack backend rewritten from scratch (better) - Revision walker rewritten from scratch (faster) - New object interdependency system (stronger) - Unique OID shortener - Reference listing In honor of one heck of a music album, released ten years ago, yesterday. --- include/git2.h | 4 ++-- wscript | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/git2.h b/include/git2.h index b08c25ed1..248c2ba39 100644 --- a/include/git2.h +++ b/include/git2.h @@ -26,9 +26,9 @@ #ifndef INCLUDE_git_git_h__ #define INCLUDE_git_git_h__ -#define LIBGIT2_VERSION "0.8.0" +#define LIBGIT2_VERSION "0.10.0" #define LIBGIT2_VER_MAJOR 0 -#define LIBGIT2_VER_MINOR 8 +#define LIBGIT2_VER_MINOR 10 #define LIBGIT2_VER_REVISION 0 #include "git2/common.h" diff --git a/wscript b/wscript index aeb17a9f4..f9daca375 100644 --- a/wscript +++ b/wscript @@ -112,10 +112,10 @@ def get_libgit2_version(git2_h): line = None with open(git2_h) as f: - line = re.search(r'^#define LIBGIT2_VERSION "(\d\.\d\.\d)"$', f.read(), re.MULTILINE) + line = re.search(r'^#define LIBGIT2_VERSION "(\d+\.\d+\.\d+)"$', f.read(), re.MULTILINE) if line is None: - raise "Failed to detect libgit2 version" + raise Exception("Failed to detect libgit2 version") return line.group(1) From 7c8a7b91431fc72a17ec579bcbca3a596071eeb1 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 15 Mar 2011 02:45:49 +0200 Subject: [PATCH 09/81] Skip the optional header in packed-refs files --- src/refs.c | 26 ++++++++++---------------- 1 file changed, 10 insertions(+), 16 deletions(-) diff --git a/src/refs.c b/src/refs.c index 1ed6b567c..b9d25a769 100644 --- a/src/refs.c +++ b/src/refs.c @@ -507,22 +507,14 @@ static int packed_load(git_repository *repo) buffer_start = (const char *)packfile.data; buffer_end = (const char *)(buffer_start) + packfile.len; - /* Does the header look like valid? */ - if (git__prefixcmp((const char *)(buffer_start), GIT_PACKEDREFS_HEADER)) { - error = GIT_EPACKEDREFSCORRUPTED; - goto cleanup; - } - - /* Let's skip the header */ - buffer_start += strlen(GIT_PACKEDREFS_HEADER); - - if (*buffer_start == '\r') + while (buffer_start < buffer_end && buffer_start[0] == '#') { + buffer_start = strchr(buffer_start, '\n'); + if (buffer_start == NULL) { + error = GIT_EPACKEDREFSCORRUPTED; + goto cleanup; + } buffer_start++; - - if (*buffer_start != '\n') - return GIT_EPACKEDREFSCORRUPTED; - - buffer_start++; + } while (buffer_start < buffer_end) { reference_oid *ref = NULL; @@ -810,7 +802,9 @@ static int packed_write(git_repository *repo) if ((error = git_filebuf_open(&pack_file, pack_file_path, 0)) < GIT_SUCCESS) return error; - /* Packfiles have a header! */ + /* Packfiles have a header... apparently + * This is in fact not required, but we might as well print it + * just for kicks */ if ((error = git_filebuf_printf(&pack_file, "%s\n", GIT_PACKEDREFS_HEADER)) < GIT_SUCCESS) return error; From bd1aa741a4e7b2b83991e7bccbfe55960929af9e Mon Sep 17 00:00:00 2001 From: Ben Noordhuis Date: Mon, 14 Mar 2011 00:07:37 +0100 Subject: [PATCH 10/81] git_object_lookup() should also check the type if the object comes from the cache --- src/object.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/object.c b/src/object.c index fce99153b..c432c6da0 100644 --- a/src/object.c +++ b/src/object.c @@ -276,6 +276,9 @@ int git_object_lookup(git_object **object_out, git_repository *repo, const git_o object = git_hashtable_lookup(repo->objects, id); if (object != NULL) { + if (type != GIT_OBJ_ANY && type != object->source.raw.type) + return GIT_EINVALIDTYPE; + *object_out = object; object->lru = ++repo->lru_counter; object->can_free = 0; From 1ee32c6dd98809e64b4178e0260dca648e287613 Mon Sep 17 00:00:00 2001 From: Ben Noordhuis Date: Mon, 14 Mar 2011 00:33:45 +0100 Subject: [PATCH 11/81] Add test case for issue GH-86 --- tests/t09-tree.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/tests/t09-tree.c b/tests/t09-tree.c index 6bc2a84bd..dfd266d16 100644 --- a/tests/t09-tree.c +++ b/tests/t09-tree.c @@ -66,6 +66,10 @@ BEGIN_TEST(read1, "read a tree from the repository") must_be_true(git_tree_entrycount(tree) == 3); + /* GH-86: git_object_lookup() should also check the type if the object comes from the cache */ + must_be_true(git_object_lookup(&obj, repo, &id, GIT_OBJ_TREE) == 0); + must_be_true(git_object_lookup(&obj, repo, &id, GIT_OBJ_BLOB) == GIT_EINVALIDTYPE); + entry = git_tree_entry_byname(tree, "README"); must_be_true(entry != NULL); From ab6a3d3de5e869253c64c1329ff73c71c2b89209 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 15 Mar 2011 03:25:44 +0200 Subject: [PATCH 12/81] Add ZLib as a built-in dependency I don't know if this is good or bad. This lets libgit2 compile cleanly on any platforms without any external dependencies, but adds a little bit of bloat... Let's test this out and see what happens. --- CMakeLists.txt | 18 +- README.md | 10 +- deps/zlib/adler32.c | 169 ++++ deps/zlib/deflate.c | 1834 ++++++++++++++++++++++++++++++++++++++++++ deps/zlib/deflate.h | 342 ++++++++ deps/zlib/inffast.c | 340 ++++++++ deps/zlib/inffast.h | 11 + deps/zlib/inffixed.h | 94 +++ deps/zlib/inflate.c | 1480 ++++++++++++++++++++++++++++++++++ deps/zlib/inflate.h | 122 +++ deps/zlib/inftrees.c | 330 ++++++++ deps/zlib/inftrees.h | 62 ++ deps/zlib/trees.c | 1244 ++++++++++++++++++++++++++++ deps/zlib/trees.h | 128 +++ deps/zlib/zconf.h | 428 ++++++++++ deps/zlib/zlib.h | 1613 +++++++++++++++++++++++++++++++++++++ deps/zlib/zutil.c | 318 ++++++++ deps/zlib/zutil.h | 274 +++++++ wscript | 13 +- 19 files changed, 8808 insertions(+), 22 deletions(-) create mode 100644 deps/zlib/adler32.c create mode 100644 deps/zlib/deflate.c create mode 100644 deps/zlib/deflate.h create mode 100644 deps/zlib/inffast.c create mode 100644 deps/zlib/inffast.h create mode 100644 deps/zlib/inffixed.h create mode 100644 deps/zlib/inflate.c create mode 100644 deps/zlib/inflate.h create mode 100644 deps/zlib/inftrees.c create mode 100644 deps/zlib/inftrees.h create mode 100644 deps/zlib/trees.c create mode 100644 deps/zlib/trees.h create mode 100644 deps/zlib/zconf.h create mode 100644 deps/zlib/zlib.h create mode 100644 deps/zlib/zutil.c create mode 100644 deps/zlib/zutil.h diff --git a/CMakeLists.txt b/CMakeLists.txt index d96924f4f..e936bcf1d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -22,8 +22,7 @@ STRING(REGEX REPLACE "^.*LIBGIT2_VERSION \"[0-9]+\\.[0-9]+\\.([0-9]+).*$" "\\1" SET(LIBGIT2_VERSION_STRING "${LIBGIT2_VERSION_MAJOR}.${LIBGIT2_VERSION_MINOR}.${LIBGIT2_VERSION_REV}") # Find required dependencies -FIND_PACKAGE(ZLIB REQUIRED) -INCLUDE_DIRECTORIES(${ZLIB_INCLUDE_DIR} src) +INCLUDE_DIRECTORIES(deps/zlib src include) # Try finding openssl FIND_PACKAGE(OpenSSL) @@ -71,13 +70,17 @@ ENDIF () # Collect sourcefiles FILE(GLOB SRC src/*.c src/backends/*.c) +FILE(GLOB SRC_ZLIB deps/zlib/*.c) FILE(GLOB SRC_SHA1 src/block-sha1/*.c) FILE(GLOB SRC_PLAT src/unix/*.c) FILE(GLOB SRC_H include/git2/*.h) +# Do not build Zlib with GZIP support +ADD_DEFINITIONS(-DNO_GZIP) + # On Windows use specific platform sources IF (WIN32 AND NOT CYGWIN) - ADD_DEFINITIONS(-DWIN32 -D_DEBUG -D_LIB -DZLIB_WINAPI) + ADD_DEFINITIONS(-DWIN32 -D_DEBUG -D_LIB) FILE(GLOB SRC_PLAT src/win32/*.c) IF (MINGW) SET(PTHREAD_LIBRARY pthread) @@ -96,9 +99,8 @@ ELSEIF (SHA1_TYPE STREQUAL "openssl") ENDIF () # Compile and link libgit2 -INCLUDE_DIRECTORIES(src include) -ADD_LIBRARY(git2 ${SRC} ${SRC_PLAT} ${SRC_SHA1}) -TARGET_LINK_LIBRARIES(git2 ${ZLIB_LIBRARY} ${LIB_SHA1} ${PTHREAD_LIBRARY} ${SQLITE3_LIBRARIES}) +ADD_LIBRARY(git2 ${SRC} ${SRC_PLAT} ${SRC_SHA1} ${SRC_ZLIB}) +TARGET_LINK_LIBRARIES(git2 ${LIB_SHA1} ${PTHREAD_LIBRARY} ${SQLITE3_LIBRARIES}) SET_TARGET_PROPERTIES(git2 PROPERTIES VERSION ${LIBGIT2_VERSION_STRING}) SET_TARGET_PROPERTIES(git2 PROPERTIES SOVERSION ${LIBGIT2_VERSION_MAJOR}) @@ -121,8 +123,8 @@ IF (BUILD_TESTS) FILE(GLOB SRC_TEST tests/t??-*.c) - ADD_EXECUTABLE(libgit2_test tests/test_main.c tests/test_lib.c tests/test_helpers.c ${SRC} ${SRC_PLAT} ${SRC_SHA1} ${SRC_TEST}) - TARGET_LINK_LIBRARIES(libgit2_test ${ZLIB_LIBRARY} ${LIB_SHA1} ${PTHREAD_LIBRARY} ${SQLITE3_LIBRARIES}) + ADD_EXECUTABLE(libgit2_test tests/test_main.c tests/test_lib.c tests/test_helpers.c ${SRC} ${SRC_PLAT} ${SRC_SHA1} ${SRC_TEST} ${SRC_ZLIB}) + TARGET_LINK_LIBRARIES(libgit2_test ${LIB_SHA1} ${PTHREAD_LIBRARY} ${SQLITE3_LIBRARIES}) ADD_TEST(libgit2_test libgit2_test) ENDIF () diff --git a/README.md b/README.md index 5b27cfdcc..ee6d87bbb 100644 --- a/README.md +++ b/README.md @@ -34,18 +34,14 @@ libgit2 is already very usable. Building libgit2 - External dependencies ======================================== -The following libraries are required to manually build the libgit2 library: - -* zlib 1.2+ - -When building in Windows using MSVC, make sure you compile ZLib using the MSVC solution that ships in its source distribution. -Alternatively, you may download precompiled binaries from: +libgit2 builds cleanly on most platforms without any external dependencies. However, the following libraries +may be used on some platforms. * LibSSL **(optional)** libgit2 can be built using the SHA1 implementation of LibSSL-Crypto, instead of the built-in custom implementations. Performance wise, they are quite similar. -* pthreads-w32 **(required on MinGW)** +* pthreads-w32 **(required for MinGW)** Building libgit2 - Using waf ====================== diff --git a/deps/zlib/adler32.c b/deps/zlib/adler32.c new file mode 100644 index 000000000..65ad6a5ad --- /dev/null +++ b/deps/zlib/adler32.c @@ -0,0 +1,169 @@ +/* adler32.c -- compute the Adler-32 checksum of a data stream + * Copyright (C) 1995-2007 Mark Adler + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* @(#) $Id$ */ + +#include "zutil.h" + +#define local static + +local uLong adler32_combine_(uLong adler1, uLong adler2, z_off64_t len2); + +#define BASE 65521UL /* largest prime smaller than 65536 */ +#define NMAX 5552 +/* NMAX is the largest n such that 255n(n+1)/2 + (n+1)(BASE-1) <= 2^32-1 */ + +#define DO1(buf,i) {adler += (buf)[i]; sum2 += adler;} +#define DO2(buf,i) DO1(buf,i); DO1(buf,i+1); +#define DO4(buf,i) DO2(buf,i); DO2(buf,i+2); +#define DO8(buf,i) DO4(buf,i); DO4(buf,i+4); +#define DO16(buf) DO8(buf,0); DO8(buf,8); + +/* use NO_DIVIDE if your processor does not do division in hardware */ +#ifdef NO_DIVIDE +# define MOD(a) \ + do { \ + if (a >= (BASE << 16)) a -= (BASE << 16); \ + if (a >= (BASE << 15)) a -= (BASE << 15); \ + if (a >= (BASE << 14)) a -= (BASE << 14); \ + if (a >= (BASE << 13)) a -= (BASE << 13); \ + if (a >= (BASE << 12)) a -= (BASE << 12); \ + if (a >= (BASE << 11)) a -= (BASE << 11); \ + if (a >= (BASE << 10)) a -= (BASE << 10); \ + if (a >= (BASE << 9)) a -= (BASE << 9); \ + if (a >= (BASE << 8)) a -= (BASE << 8); \ + if (a >= (BASE << 7)) a -= (BASE << 7); \ + if (a >= (BASE << 6)) a -= (BASE << 6); \ + if (a >= (BASE << 5)) a -= (BASE << 5); \ + if (a >= (BASE << 4)) a -= (BASE << 4); \ + if (a >= (BASE << 3)) a -= (BASE << 3); \ + if (a >= (BASE << 2)) a -= (BASE << 2); \ + if (a >= (BASE << 1)) a -= (BASE << 1); \ + if (a >= BASE) a -= BASE; \ + } while (0) +# define MOD4(a) \ + do { \ + if (a >= (BASE << 4)) a -= (BASE << 4); \ + if (a >= (BASE << 3)) a -= (BASE << 3); \ + if (a >= (BASE << 2)) a -= (BASE << 2); \ + if (a >= (BASE << 1)) a -= (BASE << 1); \ + if (a >= BASE) a -= BASE; \ + } while (0) +#else +# define MOD(a) a %= BASE +# define MOD4(a) a %= BASE +#endif + +/* ========================================================================= */ +uLong ZEXPORT adler32(adler, buf, len) + uLong adler; + const Bytef *buf; + uInt len; +{ + unsigned long sum2; + unsigned n; + + /* split Adler-32 into component sums */ + sum2 = (adler >> 16) & 0xffff; + adler &= 0xffff; + + /* in case user likes doing a byte at a time, keep it fast */ + if (len == 1) { + adler += buf[0]; + if (adler >= BASE) + adler -= BASE; + sum2 += adler; + if (sum2 >= BASE) + sum2 -= BASE; + return adler | (sum2 << 16); + } + + /* initial Adler-32 value (deferred check for len == 1 speed) */ + if (buf == Z_NULL) + return 1L; + + /* in case short lengths are provided, keep it somewhat fast */ + if (len < 16) { + while (len--) { + adler += *buf++; + sum2 += adler; + } + if (adler >= BASE) + adler -= BASE; + MOD4(sum2); /* only added so many BASE's */ + return adler | (sum2 << 16); + } + + /* do length NMAX blocks -- requires just one modulo operation */ + while (len >= NMAX) { + len -= NMAX; + n = NMAX / 16; /* NMAX is divisible by 16 */ + do { + DO16(buf); /* 16 sums unrolled */ + buf += 16; + } while (--n); + MOD(adler); + MOD(sum2); + } + + /* do remaining bytes (less than NMAX, still just one modulo) */ + if (len) { /* avoid modulos if none remaining */ + while (len >= 16) { + len -= 16; + DO16(buf); + buf += 16; + } + while (len--) { + adler += *buf++; + sum2 += adler; + } + MOD(adler); + MOD(sum2); + } + + /* return recombined sums */ + return adler | (sum2 << 16); +} + +/* ========================================================================= */ +local uLong adler32_combine_(adler1, adler2, len2) + uLong adler1; + uLong adler2; + z_off64_t len2; +{ + unsigned long sum1; + unsigned long sum2; + unsigned rem; + + /* the derivation of this formula is left as an exercise for the reader */ + rem = (unsigned)(len2 % BASE); + sum1 = adler1 & 0xffff; + sum2 = rem * sum1; + MOD(sum2); + sum1 += (adler2 & 0xffff) + BASE - 1; + sum2 += ((adler1 >> 16) & 0xffff) + ((adler2 >> 16) & 0xffff) + BASE - rem; + if (sum1 >= BASE) sum1 -= BASE; + if (sum1 >= BASE) sum1 -= BASE; + if (sum2 >= (BASE << 1)) sum2 -= (BASE << 1); + if (sum2 >= BASE) sum2 -= BASE; + return sum1 | (sum2 << 16); +} + +/* ========================================================================= */ +uLong ZEXPORT adler32_combine(adler1, adler2, len2) + uLong adler1; + uLong adler2; + z_off_t len2; +{ + return adler32_combine_(adler1, adler2, len2); +} + +uLong ZEXPORT adler32_combine64(adler1, adler2, len2) + uLong adler1; + uLong adler2; + z_off64_t len2; +{ + return adler32_combine_(adler1, adler2, len2); +} diff --git a/deps/zlib/deflate.c b/deps/zlib/deflate.c new file mode 100644 index 000000000..5c4022f3d --- /dev/null +++ b/deps/zlib/deflate.c @@ -0,0 +1,1834 @@ +/* deflate.c -- compress data using the deflation algorithm + * Copyright (C) 1995-2010 Jean-loup Gailly and Mark Adler + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* + * ALGORITHM + * + * The "deflation" process depends on being able to identify portions + * of the input text which are identical to earlier input (within a + * sliding window trailing behind the input currently being processed). + * + * The most straightforward technique turns out to be the fastest for + * most input files: try all possible matches and select the longest. + * The key feature of this algorithm is that insertions into the string + * dictionary are very simple and thus fast, and deletions are avoided + * completely. Insertions are performed at each input character, whereas + * string matches are performed only when the previous match ends. So it + * is preferable to spend more time in matches to allow very fast string + * insertions and avoid deletions. The matching algorithm for small + * strings is inspired from that of Rabin & Karp. A brute force approach + * is used to find longer strings when a small match has been found. + * A similar algorithm is used in comic (by Jan-Mark Wams) and freeze + * (by Leonid Broukhis). + * A previous version of this file used a more sophisticated algorithm + * (by Fiala and Greene) which is guaranteed to run in linear amortized + * time, but has a larger average cost, uses more memory and is patented. + * However the F&G algorithm may be faster for some highly redundant + * files if the parameter max_chain_length (described below) is too large. + * + * ACKNOWLEDGEMENTS + * + * The idea of lazy evaluation of matches is due to Jan-Mark Wams, and + * I found it in 'freeze' written by Leonid Broukhis. + * Thanks to many people for bug reports and testing. + * + * REFERENCES + * + * Deutsch, L.P.,"DEFLATE Compressed Data Format Specification". + * Available in http://www.ietf.org/rfc/rfc1951.txt + * + * A description of the Rabin and Karp algorithm is given in the book + * "Algorithms" by R. Sedgewick, Addison-Wesley, p252. + * + * Fiala,E.R., and Greene,D.H. + * Data Compression with Finite Windows, Comm.ACM, 32,4 (1989) 490-595 + * + */ + +/* @(#) $Id$ */ + +#include "deflate.h" + +const char deflate_copyright[] = + " deflate 1.2.5 Copyright 1995-2010 Jean-loup Gailly and Mark Adler "; +/* + If you use the zlib library in a product, an acknowledgment is welcome + in the documentation of your product. If for some reason you cannot + include such an acknowledgment, I would appreciate that you keep this + copyright string in the executable of your product. + */ + +/* =========================================================================== + * Function prototypes. + */ +typedef enum { + need_more, /* block not completed, need more input or more output */ + block_done, /* block flush performed */ + finish_started, /* finish started, need only more output at next deflate */ + finish_done /* finish done, accept no more input or output */ +} block_state; + +typedef block_state (*compress_func) OF((deflate_state *s, int flush)); +/* Compression function. Returns the block state after the call. */ + +local void fill_window OF((deflate_state *s)); +local block_state deflate_stored OF((deflate_state *s, int flush)); +local block_state deflate_fast OF((deflate_state *s, int flush)); +#ifndef FASTEST +local block_state deflate_slow OF((deflate_state *s, int flush)); +#endif +local block_state deflate_rle OF((deflate_state *s, int flush)); +local block_state deflate_huff OF((deflate_state *s, int flush)); +local void lm_init OF((deflate_state *s)); +local void putShortMSB OF((deflate_state *s, uInt b)); +local void flush_pending OF((z_streamp strm)); +local int read_buf OF((z_streamp strm, Bytef *buf, unsigned size)); +#ifdef ASMV + void match_init OF((void)); /* asm code initialization */ + uInt longest_match OF((deflate_state *s, IPos cur_match)); +#else +local uInt longest_match OF((deflate_state *s, IPos cur_match)); +#endif + +#ifdef DEBUG +local void check_match OF((deflate_state *s, IPos start, IPos match, + int length)); +#endif + +/* =========================================================================== + * Local data + */ + +#define NIL 0 +/* Tail of hash chains */ + +#ifndef TOO_FAR +# define TOO_FAR 4096 +#endif +/* Matches of length 3 are discarded if their distance exceeds TOO_FAR */ + +/* Values for max_lazy_match, good_match and max_chain_length, depending on + * the desired pack level (0..9). The values given below have been tuned to + * exclude worst case performance for pathological files. Better values may be + * found for specific files. + */ +typedef struct config_s { + ush good_length; /* reduce lazy search above this match length */ + ush max_lazy; /* do not perform lazy search above this match length */ + ush nice_length; /* quit search above this match length */ + ush max_chain; + compress_func func; +} config; + +#ifdef FASTEST +local const config configuration_table[2] = { +/* good lazy nice chain */ +/* 0 */ {0, 0, 0, 0, deflate_stored}, /* store only */ +/* 1 */ {4, 4, 8, 4, deflate_fast}}; /* max speed, no lazy matches */ +#else +local const config configuration_table[10] = { +/* good lazy nice chain */ +/* 0 */ {0, 0, 0, 0, deflate_stored}, /* store only */ +/* 1 */ {4, 4, 8, 4, deflate_fast}, /* max speed, no lazy matches */ +/* 2 */ {4, 5, 16, 8, deflate_fast}, +/* 3 */ {4, 6, 32, 32, deflate_fast}, + +/* 4 */ {4, 4, 16, 16, deflate_slow}, /* lazy matches */ +/* 5 */ {8, 16, 32, 32, deflate_slow}, +/* 6 */ {8, 16, 128, 128, deflate_slow}, +/* 7 */ {8, 32, 128, 256, deflate_slow}, +/* 8 */ {32, 128, 258, 1024, deflate_slow}, +/* 9 */ {32, 258, 258, 4096, deflate_slow}}; /* max compression */ +#endif + +/* Note: the deflate() code requires max_lazy >= MIN_MATCH and max_chain >= 4 + * For deflate_fast() (levels <= 3) good is ignored and lazy has a different + * meaning. + */ + +#define EQUAL 0 +/* result of memcmp for equal strings */ + +#ifndef NO_DUMMY_DECL +struct static_tree_desc_s {int dummy;}; /* for buggy compilers */ +#endif + +/* =========================================================================== + * Update a hash value with the given input byte + * IN assertion: all calls to to UPDATE_HASH are made with consecutive + * input characters, so that a running hash key can be computed from the + * previous key instead of complete recalculation each time. + */ +#define UPDATE_HASH(s,h,c) (h = (((h)<hash_shift) ^ (c)) & s->hash_mask) + + +/* =========================================================================== + * Insert string str in the dictionary and set match_head to the previous head + * of the hash chain (the most recent string with same hash key). Return + * the previous length of the hash chain. + * If this file is compiled with -DFASTEST, the compression level is forced + * to 1, and no hash chains are maintained. + * IN assertion: all calls to to INSERT_STRING are made with consecutive + * input characters and the first MIN_MATCH bytes of str are valid + * (except for the last MIN_MATCH-1 bytes of the input file). + */ +#ifdef FASTEST +#define INSERT_STRING(s, str, match_head) \ + (UPDATE_HASH(s, s->ins_h, s->window[(str) + (MIN_MATCH-1)]), \ + match_head = s->head[s->ins_h], \ + s->head[s->ins_h] = (Pos)(str)) +#else +#define INSERT_STRING(s, str, match_head) \ + (UPDATE_HASH(s, s->ins_h, s->window[(str) + (MIN_MATCH-1)]), \ + match_head = s->prev[(str) & s->w_mask] = s->head[s->ins_h], \ + s->head[s->ins_h] = (Pos)(str)) +#endif + +/* =========================================================================== + * Initialize the hash table (avoiding 64K overflow for 16 bit systems). + * prev[] will be initialized on the fly. + */ +#define CLEAR_HASH(s) \ + s->head[s->hash_size-1] = NIL; \ + zmemzero((Bytef *)s->head, (unsigned)(s->hash_size-1)*sizeof(*s->head)); + +/* ========================================================================= */ +int ZEXPORT deflateInit_(strm, level, version, stream_size) + z_streamp strm; + int level; + const char *version; + int stream_size; +{ + return deflateInit2_(strm, level, Z_DEFLATED, MAX_WBITS, DEF_MEM_LEVEL, + Z_DEFAULT_STRATEGY, version, stream_size); + /* To do: ignore strm->next_in if we use it as window */ +} + +/* ========================================================================= */ +int ZEXPORT deflateInit2_(strm, level, method, windowBits, memLevel, strategy, + version, stream_size) + z_streamp strm; + int level; + int method; + int windowBits; + int memLevel; + int strategy; + const char *version; + int stream_size; +{ + deflate_state *s; + int wrap = 1; + static const char my_version[] = ZLIB_VERSION; + + ushf *overlay; + /* We overlay pending_buf and d_buf+l_buf. This works since the average + * output size for (length,distance) codes is <= 24 bits. + */ + + if (version == Z_NULL || version[0] != my_version[0] || + stream_size != sizeof(z_stream)) { + return Z_VERSION_ERROR; + } + if (strm == Z_NULL) return Z_STREAM_ERROR; + + strm->msg = Z_NULL; + if (strm->zalloc == (alloc_func)0) { + strm->zalloc = zcalloc; + strm->opaque = (voidpf)0; + } + if (strm->zfree == (free_func)0) strm->zfree = zcfree; + +#ifdef FASTEST + if (level != 0) level = 1; +#else + if (level == Z_DEFAULT_COMPRESSION) level = 6; +#endif + + if (windowBits < 0) { /* suppress zlib wrapper */ + wrap = 0; + windowBits = -windowBits; + } +#ifdef GZIP + else if (windowBits > 15) { + wrap = 2; /* write gzip wrapper instead */ + windowBits -= 16; + } +#endif + if (memLevel < 1 || memLevel > MAX_MEM_LEVEL || method != Z_DEFLATED || + windowBits < 8 || windowBits > 15 || level < 0 || level > 9 || + strategy < 0 || strategy > Z_FIXED) { + return Z_STREAM_ERROR; + } + if (windowBits == 8) windowBits = 9; /* until 256-byte window bug fixed */ + s = (deflate_state *) ZALLOC(strm, 1, sizeof(deflate_state)); + if (s == Z_NULL) return Z_MEM_ERROR; + strm->state = (struct internal_state FAR *)s; + s->strm = strm; + + s->wrap = wrap; + s->gzhead = Z_NULL; + s->w_bits = windowBits; + s->w_size = 1 << s->w_bits; + s->w_mask = s->w_size - 1; + + s->hash_bits = memLevel + 7; + s->hash_size = 1 << s->hash_bits; + s->hash_mask = s->hash_size - 1; + s->hash_shift = ((s->hash_bits+MIN_MATCH-1)/MIN_MATCH); + + s->window = (Bytef *) ZALLOC(strm, s->w_size, 2*sizeof(Byte)); + s->prev = (Posf *) ZALLOC(strm, s->w_size, sizeof(Pos)); + s->head = (Posf *) ZALLOC(strm, s->hash_size, sizeof(Pos)); + + s->high_water = 0; /* nothing written to s->window yet */ + + s->lit_bufsize = 1 << (memLevel + 6); /* 16K elements by default */ + + overlay = (ushf *) ZALLOC(strm, s->lit_bufsize, sizeof(ush)+2); + s->pending_buf = (uchf *) overlay; + s->pending_buf_size = (ulg)s->lit_bufsize * (sizeof(ush)+2L); + + if (s->window == Z_NULL || s->prev == Z_NULL || s->head == Z_NULL || + s->pending_buf == Z_NULL) { + s->status = FINISH_STATE; + strm->msg = (char*)ERR_MSG(Z_MEM_ERROR); + deflateEnd (strm); + return Z_MEM_ERROR; + } + s->d_buf = overlay + s->lit_bufsize/sizeof(ush); + s->l_buf = s->pending_buf + (1+sizeof(ush))*s->lit_bufsize; + + s->level = level; + s->strategy = strategy; + s->method = (Byte)method; + + return deflateReset(strm); +} + +/* ========================================================================= */ +int ZEXPORT deflateSetDictionary (strm, dictionary, dictLength) + z_streamp strm; + const Bytef *dictionary; + uInt dictLength; +{ + deflate_state *s; + uInt length = dictLength; + uInt n; + IPos hash_head = 0; + + if (strm == Z_NULL || strm->state == Z_NULL || dictionary == Z_NULL || + strm->state->wrap == 2 || + (strm->state->wrap == 1 && strm->state->status != INIT_STATE)) + return Z_STREAM_ERROR; + + s = strm->state; + if (s->wrap) + strm->adler = adler32(strm->adler, dictionary, dictLength); + + if (length < MIN_MATCH) return Z_OK; + if (length > s->w_size) { + length = s->w_size; + dictionary += dictLength - length; /* use the tail of the dictionary */ + } + zmemcpy(s->window, dictionary, length); + s->strstart = length; + s->block_start = (long)length; + + /* Insert all strings in the hash table (except for the last two bytes). + * s->lookahead stays null, so s->ins_h will be recomputed at the next + * call of fill_window. + */ + s->ins_h = s->window[0]; + UPDATE_HASH(s, s->ins_h, s->window[1]); + for (n = 0; n <= length - MIN_MATCH; n++) { + INSERT_STRING(s, n, hash_head); + } + if (hash_head) hash_head = 0; /* to make compiler happy */ + return Z_OK; +} + +/* ========================================================================= */ +int ZEXPORT deflateReset (strm) + z_streamp strm; +{ + deflate_state *s; + + if (strm == Z_NULL || strm->state == Z_NULL || + strm->zalloc == (alloc_func)0 || strm->zfree == (free_func)0) { + return Z_STREAM_ERROR; + } + + strm->total_in = strm->total_out = 0; + strm->msg = Z_NULL; /* use zfree if we ever allocate msg dynamically */ + strm->data_type = Z_UNKNOWN; + + s = (deflate_state *)strm->state; + s->pending = 0; + s->pending_out = s->pending_buf; + + if (s->wrap < 0) { + s->wrap = -s->wrap; /* was made negative by deflate(..., Z_FINISH); */ + } + s->status = s->wrap ? INIT_STATE : BUSY_STATE; + strm->adler = +#ifdef GZIP + s->wrap == 2 ? crc32(0L, Z_NULL, 0) : +#endif + adler32(0L, Z_NULL, 0); + s->last_flush = Z_NO_FLUSH; + + _tr_init(s); + lm_init(s); + + return Z_OK; +} + +/* ========================================================================= */ +int ZEXPORT deflateSetHeader (strm, head) + z_streamp strm; + gz_headerp head; +{ + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + if (strm->state->wrap != 2) return Z_STREAM_ERROR; + strm->state->gzhead = head; + return Z_OK; +} + +/* ========================================================================= */ +int ZEXPORT deflatePrime (strm, bits, value) + z_streamp strm; + int bits; + int value; +{ + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + strm->state->bi_valid = bits; + strm->state->bi_buf = (ush)(value & ((1 << bits) - 1)); + return Z_OK; +} + +/* ========================================================================= */ +int ZEXPORT deflateParams(strm, level, strategy) + z_streamp strm; + int level; + int strategy; +{ + deflate_state *s; + compress_func func; + int err = Z_OK; + + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + s = strm->state; + +#ifdef FASTEST + if (level != 0) level = 1; +#else + if (level == Z_DEFAULT_COMPRESSION) level = 6; +#endif + if (level < 0 || level > 9 || strategy < 0 || strategy > Z_FIXED) { + return Z_STREAM_ERROR; + } + func = configuration_table[s->level].func; + + if ((strategy != s->strategy || func != configuration_table[level].func) && + strm->total_in != 0) { + /* Flush the last buffer: */ + err = deflate(strm, Z_BLOCK); + } + if (s->level != level) { + s->level = level; + s->max_lazy_match = configuration_table[level].max_lazy; + s->good_match = configuration_table[level].good_length; + s->nice_match = configuration_table[level].nice_length; + s->max_chain_length = configuration_table[level].max_chain; + } + s->strategy = strategy; + return err; +} + +/* ========================================================================= */ +int ZEXPORT deflateTune(strm, good_length, max_lazy, nice_length, max_chain) + z_streamp strm; + int good_length; + int max_lazy; + int nice_length; + int max_chain; +{ + deflate_state *s; + + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + s = strm->state; + s->good_match = good_length; + s->max_lazy_match = max_lazy; + s->nice_match = nice_length; + s->max_chain_length = max_chain; + return Z_OK; +} + +/* ========================================================================= + * For the default windowBits of 15 and memLevel of 8, this function returns + * a close to exact, as well as small, upper bound on the compressed size. + * They are coded as constants here for a reason--if the #define's are + * changed, then this function needs to be changed as well. The return + * value for 15 and 8 only works for those exact settings. + * + * For any setting other than those defaults for windowBits and memLevel, + * the value returned is a conservative worst case for the maximum expansion + * resulting from using fixed blocks instead of stored blocks, which deflate + * can emit on compressed data for some combinations of the parameters. + * + * This function could be more sophisticated to provide closer upper bounds for + * every combination of windowBits and memLevel. But even the conservative + * upper bound of about 14% expansion does not seem onerous for output buffer + * allocation. + */ +uLong ZEXPORT deflateBound(strm, sourceLen) + z_streamp strm; + uLong sourceLen; +{ + deflate_state *s; + uLong complen, wraplen; + Bytef *str; + + /* conservative upper bound for compressed data */ + complen = sourceLen + + ((sourceLen + 7) >> 3) + ((sourceLen + 63) >> 6) + 5; + + /* if can't get parameters, return conservative bound plus zlib wrapper */ + if (strm == Z_NULL || strm->state == Z_NULL) + return complen + 6; + + /* compute wrapper length */ + s = strm->state; + switch (s->wrap) { + case 0: /* raw deflate */ + wraplen = 0; + break; + case 1: /* zlib wrapper */ + wraplen = 6 + (s->strstart ? 4 : 0); + break; + case 2: /* gzip wrapper */ + wraplen = 18; + if (s->gzhead != Z_NULL) { /* user-supplied gzip header */ + if (s->gzhead->extra != Z_NULL) + wraplen += 2 + s->gzhead->extra_len; + str = s->gzhead->name; + if (str != Z_NULL) + do { + wraplen++; + } while (*str++); + str = s->gzhead->comment; + if (str != Z_NULL) + do { + wraplen++; + } while (*str++); + if (s->gzhead->hcrc) + wraplen += 2; + } + break; + default: /* for compiler happiness */ + wraplen = 6; + } + + /* if not default parameters, return conservative bound */ + if (s->w_bits != 15 || s->hash_bits != 8 + 7) + return complen + wraplen; + + /* default settings: return tight bound for that case */ + return sourceLen + (sourceLen >> 12) + (sourceLen >> 14) + + (sourceLen >> 25) + 13 - 6 + wraplen; +} + +/* ========================================================================= + * Put a short in the pending buffer. The 16-bit value is put in MSB order. + * IN assertion: the stream state is correct and there is enough room in + * pending_buf. + */ +local void putShortMSB (s, b) + deflate_state *s; + uInt b; +{ + put_byte(s, (Byte)(b >> 8)); + put_byte(s, (Byte)(b & 0xff)); +} + +/* ========================================================================= + * Flush as much pending output as possible. All deflate() output goes + * through this function so some applications may wish to modify it + * to avoid allocating a large strm->next_out buffer and copying into it. + * (See also read_buf()). + */ +local void flush_pending(strm) + z_streamp strm; +{ + unsigned len = strm->state->pending; + + if (len > strm->avail_out) len = strm->avail_out; + if (len == 0) return; + + zmemcpy(strm->next_out, strm->state->pending_out, len); + strm->next_out += len; + strm->state->pending_out += len; + strm->total_out += len; + strm->avail_out -= len; + strm->state->pending -= len; + if (strm->state->pending == 0) { + strm->state->pending_out = strm->state->pending_buf; + } +} + +/* ========================================================================= */ +int ZEXPORT deflate (strm, flush) + z_streamp strm; + int flush; +{ + int old_flush; /* value of flush param for previous deflate call */ + deflate_state *s; + + if (strm == Z_NULL || strm->state == Z_NULL || + flush > Z_BLOCK || flush < 0) { + return Z_STREAM_ERROR; + } + s = strm->state; + + if (strm->next_out == Z_NULL || + (strm->next_in == Z_NULL && strm->avail_in != 0) || + (s->status == FINISH_STATE && flush != Z_FINISH)) { + ERR_RETURN(strm, Z_STREAM_ERROR); + } + if (strm->avail_out == 0) ERR_RETURN(strm, Z_BUF_ERROR); + + s->strm = strm; /* just in case */ + old_flush = s->last_flush; + s->last_flush = flush; + + /* Write the header */ + if (s->status == INIT_STATE) { +#ifdef GZIP + if (s->wrap == 2) { + strm->adler = crc32(0L, Z_NULL, 0); + put_byte(s, 31); + put_byte(s, 139); + put_byte(s, 8); + if (s->gzhead == Z_NULL) { + put_byte(s, 0); + put_byte(s, 0); + put_byte(s, 0); + put_byte(s, 0); + put_byte(s, 0); + put_byte(s, s->level == 9 ? 2 : + (s->strategy >= Z_HUFFMAN_ONLY || s->level < 2 ? + 4 : 0)); + put_byte(s, OS_CODE); + s->status = BUSY_STATE; + } + else { + put_byte(s, (s->gzhead->text ? 1 : 0) + + (s->gzhead->hcrc ? 2 : 0) + + (s->gzhead->extra == Z_NULL ? 0 : 4) + + (s->gzhead->name == Z_NULL ? 0 : 8) + + (s->gzhead->comment == Z_NULL ? 0 : 16) + ); + put_byte(s, (Byte)(s->gzhead->time & 0xff)); + put_byte(s, (Byte)((s->gzhead->time >> 8) & 0xff)); + put_byte(s, (Byte)((s->gzhead->time >> 16) & 0xff)); + put_byte(s, (Byte)((s->gzhead->time >> 24) & 0xff)); + put_byte(s, s->level == 9 ? 2 : + (s->strategy >= Z_HUFFMAN_ONLY || s->level < 2 ? + 4 : 0)); + put_byte(s, s->gzhead->os & 0xff); + if (s->gzhead->extra != Z_NULL) { + put_byte(s, s->gzhead->extra_len & 0xff); + put_byte(s, (s->gzhead->extra_len >> 8) & 0xff); + } + if (s->gzhead->hcrc) + strm->adler = crc32(strm->adler, s->pending_buf, + s->pending); + s->gzindex = 0; + s->status = EXTRA_STATE; + } + } + else +#endif + { + uInt header = (Z_DEFLATED + ((s->w_bits-8)<<4)) << 8; + uInt level_flags; + + if (s->strategy >= Z_HUFFMAN_ONLY || s->level < 2) + level_flags = 0; + else if (s->level < 6) + level_flags = 1; + else if (s->level == 6) + level_flags = 2; + else + level_flags = 3; + header |= (level_flags << 6); + if (s->strstart != 0) header |= PRESET_DICT; + header += 31 - (header % 31); + + s->status = BUSY_STATE; + putShortMSB(s, header); + + /* Save the adler32 of the preset dictionary: */ + if (s->strstart != 0) { + putShortMSB(s, (uInt)(strm->adler >> 16)); + putShortMSB(s, (uInt)(strm->adler & 0xffff)); + } + strm->adler = adler32(0L, Z_NULL, 0); + } + } +#ifdef GZIP + if (s->status == EXTRA_STATE) { + if (s->gzhead->extra != Z_NULL) { + uInt beg = s->pending; /* start of bytes to update crc */ + + while (s->gzindex < (s->gzhead->extra_len & 0xffff)) { + if (s->pending == s->pending_buf_size) { + if (s->gzhead->hcrc && s->pending > beg) + strm->adler = crc32(strm->adler, s->pending_buf + beg, + s->pending - beg); + flush_pending(strm); + beg = s->pending; + if (s->pending == s->pending_buf_size) + break; + } + put_byte(s, s->gzhead->extra[s->gzindex]); + s->gzindex++; + } + if (s->gzhead->hcrc && s->pending > beg) + strm->adler = crc32(strm->adler, s->pending_buf + beg, + s->pending - beg); + if (s->gzindex == s->gzhead->extra_len) { + s->gzindex = 0; + s->status = NAME_STATE; + } + } + else + s->status = NAME_STATE; + } + if (s->status == NAME_STATE) { + if (s->gzhead->name != Z_NULL) { + uInt beg = s->pending; /* start of bytes to update crc */ + int val; + + do { + if (s->pending == s->pending_buf_size) { + if (s->gzhead->hcrc && s->pending > beg) + strm->adler = crc32(strm->adler, s->pending_buf + beg, + s->pending - beg); + flush_pending(strm); + beg = s->pending; + if (s->pending == s->pending_buf_size) { + val = 1; + break; + } + } + val = s->gzhead->name[s->gzindex++]; + put_byte(s, val); + } while (val != 0); + if (s->gzhead->hcrc && s->pending > beg) + strm->adler = crc32(strm->adler, s->pending_buf + beg, + s->pending - beg); + if (val == 0) { + s->gzindex = 0; + s->status = COMMENT_STATE; + } + } + else + s->status = COMMENT_STATE; + } + if (s->status == COMMENT_STATE) { + if (s->gzhead->comment != Z_NULL) { + uInt beg = s->pending; /* start of bytes to update crc */ + int val; + + do { + if (s->pending == s->pending_buf_size) { + if (s->gzhead->hcrc && s->pending > beg) + strm->adler = crc32(strm->adler, s->pending_buf + beg, + s->pending - beg); + flush_pending(strm); + beg = s->pending; + if (s->pending == s->pending_buf_size) { + val = 1; + break; + } + } + val = s->gzhead->comment[s->gzindex++]; + put_byte(s, val); + } while (val != 0); + if (s->gzhead->hcrc && s->pending > beg) + strm->adler = crc32(strm->adler, s->pending_buf + beg, + s->pending - beg); + if (val == 0) + s->status = HCRC_STATE; + } + else + s->status = HCRC_STATE; + } + if (s->status == HCRC_STATE) { + if (s->gzhead->hcrc) { + if (s->pending + 2 > s->pending_buf_size) + flush_pending(strm); + if (s->pending + 2 <= s->pending_buf_size) { + put_byte(s, (Byte)(strm->adler & 0xff)); + put_byte(s, (Byte)((strm->adler >> 8) & 0xff)); + strm->adler = crc32(0L, Z_NULL, 0); + s->status = BUSY_STATE; + } + } + else + s->status = BUSY_STATE; + } +#endif + + /* Flush as much pending output as possible */ + if (s->pending != 0) { + flush_pending(strm); + if (strm->avail_out == 0) { + /* Since avail_out is 0, deflate will be called again with + * more output space, but possibly with both pending and + * avail_in equal to zero. There won't be anything to do, + * but this is not an error situation so make sure we + * return OK instead of BUF_ERROR at next call of deflate: + */ + s->last_flush = -1; + return Z_OK; + } + + /* Make sure there is something to do and avoid duplicate consecutive + * flushes. For repeated and useless calls with Z_FINISH, we keep + * returning Z_STREAM_END instead of Z_BUF_ERROR. + */ + } else if (strm->avail_in == 0 && flush <= old_flush && + flush != Z_FINISH) { + ERR_RETURN(strm, Z_BUF_ERROR); + } + + /* User must not provide more input after the first FINISH: */ + if (s->status == FINISH_STATE && strm->avail_in != 0) { + ERR_RETURN(strm, Z_BUF_ERROR); + } + + /* Start a new block or continue the current one. + */ + if (strm->avail_in != 0 || s->lookahead != 0 || + (flush != Z_NO_FLUSH && s->status != FINISH_STATE)) { + block_state bstate; + + bstate = s->strategy == Z_HUFFMAN_ONLY ? deflate_huff(s, flush) : + (s->strategy == Z_RLE ? deflate_rle(s, flush) : + (*(configuration_table[s->level].func))(s, flush)); + + if (bstate == finish_started || bstate == finish_done) { + s->status = FINISH_STATE; + } + if (bstate == need_more || bstate == finish_started) { + if (strm->avail_out == 0) { + s->last_flush = -1; /* avoid BUF_ERROR next call, see above */ + } + return Z_OK; + /* If flush != Z_NO_FLUSH && avail_out == 0, the next call + * of deflate should use the same flush parameter to make sure + * that the flush is complete. So we don't have to output an + * empty block here, this will be done at next call. This also + * ensures that for a very small output buffer, we emit at most + * one empty block. + */ + } + if (bstate == block_done) { + if (flush == Z_PARTIAL_FLUSH) { + _tr_align(s); + } else if (flush != Z_BLOCK) { /* FULL_FLUSH or SYNC_FLUSH */ + _tr_stored_block(s, (char*)0, 0L, 0); + /* For a full flush, this empty block will be recognized + * as a special marker by inflate_sync(). + */ + if (flush == Z_FULL_FLUSH) { + CLEAR_HASH(s); /* forget history */ + if (s->lookahead == 0) { + s->strstart = 0; + s->block_start = 0L; + } + } + } + flush_pending(strm); + if (strm->avail_out == 0) { + s->last_flush = -1; /* avoid BUF_ERROR at next call, see above */ + return Z_OK; + } + } + } + Assert(strm->avail_out > 0, "bug2"); + + if (flush != Z_FINISH) return Z_OK; + if (s->wrap <= 0) return Z_STREAM_END; + + /* Write the trailer */ +#ifdef GZIP + if (s->wrap == 2) { + put_byte(s, (Byte)(strm->adler & 0xff)); + put_byte(s, (Byte)((strm->adler >> 8) & 0xff)); + put_byte(s, (Byte)((strm->adler >> 16) & 0xff)); + put_byte(s, (Byte)((strm->adler >> 24) & 0xff)); + put_byte(s, (Byte)(strm->total_in & 0xff)); + put_byte(s, (Byte)((strm->total_in >> 8) & 0xff)); + put_byte(s, (Byte)((strm->total_in >> 16) & 0xff)); + put_byte(s, (Byte)((strm->total_in >> 24) & 0xff)); + } + else +#endif + { + putShortMSB(s, (uInt)(strm->adler >> 16)); + putShortMSB(s, (uInt)(strm->adler & 0xffff)); + } + flush_pending(strm); + /* If avail_out is zero, the application will call deflate again + * to flush the rest. + */ + if (s->wrap > 0) s->wrap = -s->wrap; /* write the trailer only once! */ + return s->pending != 0 ? Z_OK : Z_STREAM_END; +} + +/* ========================================================================= */ +int ZEXPORT deflateEnd (strm) + z_streamp strm; +{ + int status; + + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + + status = strm->state->status; + if (status != INIT_STATE && + status != EXTRA_STATE && + status != NAME_STATE && + status != COMMENT_STATE && + status != HCRC_STATE && + status != BUSY_STATE && + status != FINISH_STATE) { + return Z_STREAM_ERROR; + } + + /* Deallocate in reverse order of allocations: */ + TRY_FREE(strm, strm->state->pending_buf); + TRY_FREE(strm, strm->state->head); + TRY_FREE(strm, strm->state->prev); + TRY_FREE(strm, strm->state->window); + + ZFREE(strm, strm->state); + strm->state = Z_NULL; + + return status == BUSY_STATE ? Z_DATA_ERROR : Z_OK; +} + +/* ========================================================================= + * Copy the source state to the destination state. + * To simplify the source, this is not supported for 16-bit MSDOS (which + * doesn't have enough memory anyway to duplicate compression states). + */ +int ZEXPORT deflateCopy (dest, source) + z_streamp dest; + z_streamp source; +{ +#ifdef MAXSEG_64K + return Z_STREAM_ERROR; +#else + deflate_state *ds; + deflate_state *ss; + ushf *overlay; + + + if (source == Z_NULL || dest == Z_NULL || source->state == Z_NULL) { + return Z_STREAM_ERROR; + } + + ss = source->state; + + zmemcpy(dest, source, sizeof(z_stream)); + + ds = (deflate_state *) ZALLOC(dest, 1, sizeof(deflate_state)); + if (ds == Z_NULL) return Z_MEM_ERROR; + dest->state = (struct internal_state FAR *) ds; + zmemcpy(ds, ss, sizeof(deflate_state)); + ds->strm = dest; + + ds->window = (Bytef *) ZALLOC(dest, ds->w_size, 2*sizeof(Byte)); + ds->prev = (Posf *) ZALLOC(dest, ds->w_size, sizeof(Pos)); + ds->head = (Posf *) ZALLOC(dest, ds->hash_size, sizeof(Pos)); + overlay = (ushf *) ZALLOC(dest, ds->lit_bufsize, sizeof(ush)+2); + ds->pending_buf = (uchf *) overlay; + + if (ds->window == Z_NULL || ds->prev == Z_NULL || ds->head == Z_NULL || + ds->pending_buf == Z_NULL) { + deflateEnd (dest); + return Z_MEM_ERROR; + } + /* following zmemcpy do not work for 16-bit MSDOS */ + zmemcpy(ds->window, ss->window, ds->w_size * 2 * sizeof(Byte)); + zmemcpy(ds->prev, ss->prev, ds->w_size * sizeof(Pos)); + zmemcpy(ds->head, ss->head, ds->hash_size * sizeof(Pos)); + zmemcpy(ds->pending_buf, ss->pending_buf, (uInt)ds->pending_buf_size); + + ds->pending_out = ds->pending_buf + (ss->pending_out - ss->pending_buf); + ds->d_buf = overlay + ds->lit_bufsize/sizeof(ush); + ds->l_buf = ds->pending_buf + (1+sizeof(ush))*ds->lit_bufsize; + + ds->l_desc.dyn_tree = ds->dyn_ltree; + ds->d_desc.dyn_tree = ds->dyn_dtree; + ds->bl_desc.dyn_tree = ds->bl_tree; + + return Z_OK; +#endif /* MAXSEG_64K */ +} + +/* =========================================================================== + * Read a new buffer from the current input stream, update the adler32 + * and total number of bytes read. All deflate() input goes through + * this function so some applications may wish to modify it to avoid + * allocating a large strm->next_in buffer and copying from it. + * (See also flush_pending()). + */ +local int read_buf(strm, buf, size) + z_streamp strm; + Bytef *buf; + unsigned size; +{ + unsigned len = strm->avail_in; + + if (len > size) len = size; + if (len == 0) return 0; + + strm->avail_in -= len; + + if (strm->state->wrap == 1) { + strm->adler = adler32(strm->adler, strm->next_in, len); + } +#ifdef GZIP + else if (strm->state->wrap == 2) { + strm->adler = crc32(strm->adler, strm->next_in, len); + } +#endif + zmemcpy(buf, strm->next_in, len); + strm->next_in += len; + strm->total_in += len; + + return (int)len; +} + +/* =========================================================================== + * Initialize the "longest match" routines for a new zlib stream + */ +local void lm_init (s) + deflate_state *s; +{ + s->window_size = (ulg)2L*s->w_size; + + CLEAR_HASH(s); + + /* Set the default configuration parameters: + */ + s->max_lazy_match = configuration_table[s->level].max_lazy; + s->good_match = configuration_table[s->level].good_length; + s->nice_match = configuration_table[s->level].nice_length; + s->max_chain_length = configuration_table[s->level].max_chain; + + s->strstart = 0; + s->block_start = 0L; + s->lookahead = 0; + s->match_length = s->prev_length = MIN_MATCH-1; + s->match_available = 0; + s->ins_h = 0; +#ifndef FASTEST +#ifdef ASMV + match_init(); /* initialize the asm code */ +#endif +#endif +} + +#ifndef FASTEST +/* =========================================================================== + * Set match_start to the longest match starting at the given string and + * return its length. Matches shorter or equal to prev_length are discarded, + * in which case the result is equal to prev_length and match_start is + * garbage. + * IN assertions: cur_match is the head of the hash chain for the current + * string (strstart) and its distance is <= MAX_DIST, and prev_length >= 1 + * OUT assertion: the match length is not greater than s->lookahead. + */ +#ifndef ASMV +/* For 80x86 and 680x0, an optimized version will be provided in match.asm or + * match.S. The code will be functionally equivalent. + */ +local uInt longest_match(s, cur_match) + deflate_state *s; + IPos cur_match; /* current match */ +{ + unsigned chain_length = s->max_chain_length;/* max hash chain length */ + register Bytef *scan = s->window + s->strstart; /* current string */ + register Bytef *match; /* matched string */ + register int len; /* length of current match */ + int best_len = s->prev_length; /* best match length so far */ + int nice_match = s->nice_match; /* stop if match long enough */ + IPos limit = s->strstart > (IPos)MAX_DIST(s) ? + s->strstart - (IPos)MAX_DIST(s) : NIL; + /* Stop when cur_match becomes <= limit. To simplify the code, + * we prevent matches with the string of window index 0. + */ + Posf *prev = s->prev; + uInt wmask = s->w_mask; + +#ifdef UNALIGNED_OK + /* Compare two bytes at a time. Note: this is not always beneficial. + * Try with and without -DUNALIGNED_OK to check. + */ + register Bytef *strend = s->window + s->strstart + MAX_MATCH - 1; + register ush scan_start = *(ushf*)scan; + register ush scan_end = *(ushf*)(scan+best_len-1); +#else + register Bytef *strend = s->window + s->strstart + MAX_MATCH; + register Byte scan_end1 = scan[best_len-1]; + register Byte scan_end = scan[best_len]; +#endif + + /* The code is optimized for HASH_BITS >= 8 and MAX_MATCH-2 multiple of 16. + * It is easy to get rid of this optimization if necessary. + */ + Assert(s->hash_bits >= 8 && MAX_MATCH == 258, "Code too clever"); + + /* Do not waste too much time if we already have a good match: */ + if (s->prev_length >= s->good_match) { + chain_length >>= 2; + } + /* Do not look for matches beyond the end of the input. This is necessary + * to make deflate deterministic. + */ + if ((uInt)nice_match > s->lookahead) nice_match = s->lookahead; + + Assert((ulg)s->strstart <= s->window_size-MIN_LOOKAHEAD, "need lookahead"); + + do { + Assert(cur_match < s->strstart, "no future"); + match = s->window + cur_match; + + /* Skip to next match if the match length cannot increase + * or if the match length is less than 2. Note that the checks below + * for insufficient lookahead only occur occasionally for performance + * reasons. Therefore uninitialized memory will be accessed, and + * conditional jumps will be made that depend on those values. + * However the length of the match is limited to the lookahead, so + * the output of deflate is not affected by the uninitialized values. + */ +#if (defined(UNALIGNED_OK) && MAX_MATCH == 258) + /* This code assumes sizeof(unsigned short) == 2. Do not use + * UNALIGNED_OK if your compiler uses a different size. + */ + if (*(ushf*)(match+best_len-1) != scan_end || + *(ushf*)match != scan_start) continue; + + /* It is not necessary to compare scan[2] and match[2] since they are + * always equal when the other bytes match, given that the hash keys + * are equal and that HASH_BITS >= 8. Compare 2 bytes at a time at + * strstart+3, +5, ... up to strstart+257. We check for insufficient + * lookahead only every 4th comparison; the 128th check will be made + * at strstart+257. If MAX_MATCH-2 is not a multiple of 8, it is + * necessary to put more guard bytes at the end of the window, or + * to check more often for insufficient lookahead. + */ + Assert(scan[2] == match[2], "scan[2]?"); + scan++, match++; + do { + } while (*(ushf*)(scan+=2) == *(ushf*)(match+=2) && + *(ushf*)(scan+=2) == *(ushf*)(match+=2) && + *(ushf*)(scan+=2) == *(ushf*)(match+=2) && + *(ushf*)(scan+=2) == *(ushf*)(match+=2) && + scan < strend); + /* The funny "do {}" generates better code on most compilers */ + + /* Here, scan <= window+strstart+257 */ + Assert(scan <= s->window+(unsigned)(s->window_size-1), "wild scan"); + if (*scan == *match) scan++; + + len = (MAX_MATCH - 1) - (int)(strend-scan); + scan = strend - (MAX_MATCH-1); + +#else /* UNALIGNED_OK */ + + if (match[best_len] != scan_end || + match[best_len-1] != scan_end1 || + *match != *scan || + *++match != scan[1]) continue; + + /* The check at best_len-1 can be removed because it will be made + * again later. (This heuristic is not always a win.) + * It is not necessary to compare scan[2] and match[2] since they + * are always equal when the other bytes match, given that + * the hash keys are equal and that HASH_BITS >= 8. + */ + scan += 2, match++; + Assert(*scan == *match, "match[2]?"); + + /* We check for insufficient lookahead only every 8th comparison; + * the 256th check will be made at strstart+258. + */ + do { + } while (*++scan == *++match && *++scan == *++match && + *++scan == *++match && *++scan == *++match && + *++scan == *++match && *++scan == *++match && + *++scan == *++match && *++scan == *++match && + scan < strend); + + Assert(scan <= s->window+(unsigned)(s->window_size-1), "wild scan"); + + len = MAX_MATCH - (int)(strend - scan); + scan = strend - MAX_MATCH; + +#endif /* UNALIGNED_OK */ + + if (len > best_len) { + s->match_start = cur_match; + best_len = len; + if (len >= nice_match) break; +#ifdef UNALIGNED_OK + scan_end = *(ushf*)(scan+best_len-1); +#else + scan_end1 = scan[best_len-1]; + scan_end = scan[best_len]; +#endif + } + } while ((cur_match = prev[cur_match & wmask]) > limit + && --chain_length != 0); + + if ((uInt)best_len <= s->lookahead) return (uInt)best_len; + return s->lookahead; +} +#endif /* ASMV */ + +#else /* FASTEST */ + +/* --------------------------------------------------------------------------- + * Optimized version for FASTEST only + */ +local uInt longest_match(s, cur_match) + deflate_state *s; + IPos cur_match; /* current match */ +{ + register Bytef *scan = s->window + s->strstart; /* current string */ + register Bytef *match; /* matched string */ + register int len; /* length of current match */ + register Bytef *strend = s->window + s->strstart + MAX_MATCH; + + /* The code is optimized for HASH_BITS >= 8 and MAX_MATCH-2 multiple of 16. + * It is easy to get rid of this optimization if necessary. + */ + Assert(s->hash_bits >= 8 && MAX_MATCH == 258, "Code too clever"); + + Assert((ulg)s->strstart <= s->window_size-MIN_LOOKAHEAD, "need lookahead"); + + Assert(cur_match < s->strstart, "no future"); + + match = s->window + cur_match; + + /* Return failure if the match length is less than 2: + */ + if (match[0] != scan[0] || match[1] != scan[1]) return MIN_MATCH-1; + + /* The check at best_len-1 can be removed because it will be made + * again later. (This heuristic is not always a win.) + * It is not necessary to compare scan[2] and match[2] since they + * are always equal when the other bytes match, given that + * the hash keys are equal and that HASH_BITS >= 8. + */ + scan += 2, match += 2; + Assert(*scan == *match, "match[2]?"); + + /* We check for insufficient lookahead only every 8th comparison; + * the 256th check will be made at strstart+258. + */ + do { + } while (*++scan == *++match && *++scan == *++match && + *++scan == *++match && *++scan == *++match && + *++scan == *++match && *++scan == *++match && + *++scan == *++match && *++scan == *++match && + scan < strend); + + Assert(scan <= s->window+(unsigned)(s->window_size-1), "wild scan"); + + len = MAX_MATCH - (int)(strend - scan); + + if (len < MIN_MATCH) return MIN_MATCH - 1; + + s->match_start = cur_match; + return (uInt)len <= s->lookahead ? (uInt)len : s->lookahead; +} + +#endif /* FASTEST */ + +#ifdef DEBUG +/* =========================================================================== + * Check that the match at match_start is indeed a match. + */ +local void check_match(s, start, match, length) + deflate_state *s; + IPos start, match; + int length; +{ + /* check that the match is indeed a match */ + if (zmemcmp(s->window + match, + s->window + start, length) != EQUAL) { + fprintf(stderr, " start %u, match %u, length %d\n", + start, match, length); + do { + fprintf(stderr, "%c%c", s->window[match++], s->window[start++]); + } while (--length != 0); + z_error("invalid match"); + } + if (z_verbose > 1) { + fprintf(stderr,"\\[%d,%d]", start-match, length); + do { putc(s->window[start++], stderr); } while (--length != 0); + } +} +#else +# define check_match(s, start, match, length) +#endif /* DEBUG */ + +/* =========================================================================== + * Fill the window when the lookahead becomes insufficient. + * Updates strstart and lookahead. + * + * IN assertion: lookahead < MIN_LOOKAHEAD + * OUT assertions: strstart <= window_size-MIN_LOOKAHEAD + * At least one byte has been read, or avail_in == 0; reads are + * performed for at least two bytes (required for the zip translate_eol + * option -- not supported here). + */ +local void fill_window(s) + deflate_state *s; +{ + register unsigned n, m; + register Posf *p; + unsigned more; /* Amount of free space at the end of the window. */ + uInt wsize = s->w_size; + + do { + more = (unsigned)(s->window_size -(ulg)s->lookahead -(ulg)s->strstart); + + /* Deal with !@#$% 64K limit: */ + if (sizeof(int) <= 2) { + if (more == 0 && s->strstart == 0 && s->lookahead == 0) { + more = wsize; + + } else if (more == (unsigned)(-1)) { + /* Very unlikely, but possible on 16 bit machine if + * strstart == 0 && lookahead == 1 (input done a byte at time) + */ + more--; + } + } + + /* If the window is almost full and there is insufficient lookahead, + * move the upper half to the lower one to make room in the upper half. + */ + if (s->strstart >= wsize+MAX_DIST(s)) { + + zmemcpy(s->window, s->window+wsize, (unsigned)wsize); + s->match_start -= wsize; + s->strstart -= wsize; /* we now have strstart >= MAX_DIST */ + s->block_start -= (long) wsize; + + /* Slide the hash table (could be avoided with 32 bit values + at the expense of memory usage). We slide even when level == 0 + to keep the hash table consistent if we switch back to level > 0 + later. (Using level 0 permanently is not an optimal usage of + zlib, so we don't care about this pathological case.) + */ + n = s->hash_size; + p = &s->head[n]; + do { + m = *--p; + *p = (Pos)(m >= wsize ? m-wsize : NIL); + } while (--n); + + n = wsize; +#ifndef FASTEST + p = &s->prev[n]; + do { + m = *--p; + *p = (Pos)(m >= wsize ? m-wsize : NIL); + /* If n is not on any hash chain, prev[n] is garbage but + * its value will never be used. + */ + } while (--n); +#endif + more += wsize; + } + if (s->strm->avail_in == 0) return; + + /* If there was no sliding: + * strstart <= WSIZE+MAX_DIST-1 && lookahead <= MIN_LOOKAHEAD - 1 && + * more == window_size - lookahead - strstart + * => more >= window_size - (MIN_LOOKAHEAD-1 + WSIZE + MAX_DIST-1) + * => more >= window_size - 2*WSIZE + 2 + * In the BIG_MEM or MMAP case (not yet supported), + * window_size == input_size + MIN_LOOKAHEAD && + * strstart + s->lookahead <= input_size => more >= MIN_LOOKAHEAD. + * Otherwise, window_size == 2*WSIZE so more >= 2. + * If there was sliding, more >= WSIZE. So in all cases, more >= 2. + */ + Assert(more >= 2, "more < 2"); + + n = read_buf(s->strm, s->window + s->strstart + s->lookahead, more); + s->lookahead += n; + + /* Initialize the hash value now that we have some input: */ + if (s->lookahead >= MIN_MATCH) { + s->ins_h = s->window[s->strstart]; + UPDATE_HASH(s, s->ins_h, s->window[s->strstart+1]); +#if MIN_MATCH != 3 + Call UPDATE_HASH() MIN_MATCH-3 more times +#endif + } + /* If the whole input has less than MIN_MATCH bytes, ins_h is garbage, + * but this is not important since only literal bytes will be emitted. + */ + + } while (s->lookahead < MIN_LOOKAHEAD && s->strm->avail_in != 0); + + /* If the WIN_INIT bytes after the end of the current data have never been + * written, then zero those bytes in order to avoid memory check reports of + * the use of uninitialized (or uninitialised as Julian writes) bytes by + * the longest match routines. Update the high water mark for the next + * time through here. WIN_INIT is set to MAX_MATCH since the longest match + * routines allow scanning to strstart + MAX_MATCH, ignoring lookahead. + */ + if (s->high_water < s->window_size) { + ulg curr = s->strstart + (ulg)(s->lookahead); + ulg init; + + if (s->high_water < curr) { + /* Previous high water mark below current data -- zero WIN_INIT + * bytes or up to end of window, whichever is less. + */ + init = s->window_size - curr; + if (init > WIN_INIT) + init = WIN_INIT; + zmemzero(s->window + curr, (unsigned)init); + s->high_water = curr + init; + } + else if (s->high_water < (ulg)curr + WIN_INIT) { + /* High water mark at or above current data, but below current data + * plus WIN_INIT -- zero out to current data plus WIN_INIT, or up + * to end of window, whichever is less. + */ + init = (ulg)curr + WIN_INIT - s->high_water; + if (init > s->window_size - s->high_water) + init = s->window_size - s->high_water; + zmemzero(s->window + s->high_water, (unsigned)init); + s->high_water += init; + } + } +} + +/* =========================================================================== + * Flush the current block, with given end-of-file flag. + * IN assertion: strstart is set to the end of the current match. + */ +#define FLUSH_BLOCK_ONLY(s, last) { \ + _tr_flush_block(s, (s->block_start >= 0L ? \ + (charf *)&s->window[(unsigned)s->block_start] : \ + (charf *)Z_NULL), \ + (ulg)((long)s->strstart - s->block_start), \ + (last)); \ + s->block_start = s->strstart; \ + flush_pending(s->strm); \ + Tracev((stderr,"[FLUSH]")); \ +} + +/* Same but force premature exit if necessary. */ +#define FLUSH_BLOCK(s, last) { \ + FLUSH_BLOCK_ONLY(s, last); \ + if (s->strm->avail_out == 0) return (last) ? finish_started : need_more; \ +} + +/* =========================================================================== + * Copy without compression as much as possible from the input stream, return + * the current block state. + * This function does not insert new strings in the dictionary since + * uncompressible data is probably not useful. This function is used + * only for the level=0 compression option. + * NOTE: this function should be optimized to avoid extra copying from + * window to pending_buf. + */ +local block_state deflate_stored(s, flush) + deflate_state *s; + int flush; +{ + /* Stored blocks are limited to 0xffff bytes, pending_buf is limited + * to pending_buf_size, and each stored block has a 5 byte header: + */ + ulg max_block_size = 0xffff; + ulg max_start; + + if (max_block_size > s->pending_buf_size - 5) { + max_block_size = s->pending_buf_size - 5; + } + + /* Copy as much as possible from input to output: */ + for (;;) { + /* Fill the window as much as possible: */ + if (s->lookahead <= 1) { + + Assert(s->strstart < s->w_size+MAX_DIST(s) || + s->block_start >= (long)s->w_size, "slide too late"); + + fill_window(s); + if (s->lookahead == 0 && flush == Z_NO_FLUSH) return need_more; + + if (s->lookahead == 0) break; /* flush the current block */ + } + Assert(s->block_start >= 0L, "block gone"); + + s->strstart += s->lookahead; + s->lookahead = 0; + + /* Emit a stored block if pending_buf will be full: */ + max_start = s->block_start + max_block_size; + if (s->strstart == 0 || (ulg)s->strstart >= max_start) { + /* strstart == 0 is possible when wraparound on 16-bit machine */ + s->lookahead = (uInt)(s->strstart - max_start); + s->strstart = (uInt)max_start; + FLUSH_BLOCK(s, 0); + } + /* Flush if we may have to slide, otherwise block_start may become + * negative and the data will be gone: + */ + if (s->strstart - (uInt)s->block_start >= MAX_DIST(s)) { + FLUSH_BLOCK(s, 0); + } + } + FLUSH_BLOCK(s, flush == Z_FINISH); + return flush == Z_FINISH ? finish_done : block_done; +} + +/* =========================================================================== + * Compress as much as possible from the input stream, return the current + * block state. + * This function does not perform lazy evaluation of matches and inserts + * new strings in the dictionary only for unmatched strings or for short + * matches. It is used only for the fast compression options. + */ +local block_state deflate_fast(s, flush) + deflate_state *s; + int flush; +{ + IPos hash_head; /* head of the hash chain */ + int bflush; /* set if current block must be flushed */ + + for (;;) { + /* Make sure that we always have enough lookahead, except + * at the end of the input file. We need MAX_MATCH bytes + * for the next match, plus MIN_MATCH bytes to insert the + * string following the next match. + */ + if (s->lookahead < MIN_LOOKAHEAD) { + fill_window(s); + if (s->lookahead < MIN_LOOKAHEAD && flush == Z_NO_FLUSH) { + return need_more; + } + if (s->lookahead == 0) break; /* flush the current block */ + } + + /* Insert the string window[strstart .. strstart+2] in the + * dictionary, and set hash_head to the head of the hash chain: + */ + hash_head = NIL; + if (s->lookahead >= MIN_MATCH) { + INSERT_STRING(s, s->strstart, hash_head); + } + + /* Find the longest match, discarding those <= prev_length. + * At this point we have always match_length < MIN_MATCH + */ + if (hash_head != NIL && s->strstart - hash_head <= MAX_DIST(s)) { + /* To simplify the code, we prevent matches with the string + * of window index 0 (in particular we have to avoid a match + * of the string with itself at the start of the input file). + */ + s->match_length = longest_match (s, hash_head); + /* longest_match() sets match_start */ + } + if (s->match_length >= MIN_MATCH) { + check_match(s, s->strstart, s->match_start, s->match_length); + + _tr_tally_dist(s, s->strstart - s->match_start, + s->match_length - MIN_MATCH, bflush); + + s->lookahead -= s->match_length; + + /* Insert new strings in the hash table only if the match length + * is not too large. This saves time but degrades compression. + */ +#ifndef FASTEST + if (s->match_length <= s->max_insert_length && + s->lookahead >= MIN_MATCH) { + s->match_length--; /* string at strstart already in table */ + do { + s->strstart++; + INSERT_STRING(s, s->strstart, hash_head); + /* strstart never exceeds WSIZE-MAX_MATCH, so there are + * always MIN_MATCH bytes ahead. + */ + } while (--s->match_length != 0); + s->strstart++; + } else +#endif + { + s->strstart += s->match_length; + s->match_length = 0; + s->ins_h = s->window[s->strstart]; + UPDATE_HASH(s, s->ins_h, s->window[s->strstart+1]); +#if MIN_MATCH != 3 + Call UPDATE_HASH() MIN_MATCH-3 more times +#endif + /* If lookahead < MIN_MATCH, ins_h is garbage, but it does not + * matter since it will be recomputed at next deflate call. + */ + } + } else { + /* No match, output a literal byte */ + Tracevv((stderr,"%c", s->window[s->strstart])); + _tr_tally_lit (s, s->window[s->strstart], bflush); + s->lookahead--; + s->strstart++; + } + if (bflush) FLUSH_BLOCK(s, 0); + } + FLUSH_BLOCK(s, flush == Z_FINISH); + return flush == Z_FINISH ? finish_done : block_done; +} + +#ifndef FASTEST +/* =========================================================================== + * Same as above, but achieves better compression. We use a lazy + * evaluation for matches: a match is finally adopted only if there is + * no better match at the next window position. + */ +local block_state deflate_slow(s, flush) + deflate_state *s; + int flush; +{ + IPos hash_head; /* head of hash chain */ + int bflush; /* set if current block must be flushed */ + + /* Process the input block. */ + for (;;) { + /* Make sure that we always have enough lookahead, except + * at the end of the input file. We need MAX_MATCH bytes + * for the next match, plus MIN_MATCH bytes to insert the + * string following the next match. + */ + if (s->lookahead < MIN_LOOKAHEAD) { + fill_window(s); + if (s->lookahead < MIN_LOOKAHEAD && flush == Z_NO_FLUSH) { + return need_more; + } + if (s->lookahead == 0) break; /* flush the current block */ + } + + /* Insert the string window[strstart .. strstart+2] in the + * dictionary, and set hash_head to the head of the hash chain: + */ + hash_head = NIL; + if (s->lookahead >= MIN_MATCH) { + INSERT_STRING(s, s->strstart, hash_head); + } + + /* Find the longest match, discarding those <= prev_length. + */ + s->prev_length = s->match_length, s->prev_match = s->match_start; + s->match_length = MIN_MATCH-1; + + if (hash_head != NIL && s->prev_length < s->max_lazy_match && + s->strstart - hash_head <= MAX_DIST(s)) { + /* To simplify the code, we prevent matches with the string + * of window index 0 (in particular we have to avoid a match + * of the string with itself at the start of the input file). + */ + s->match_length = longest_match (s, hash_head); + /* longest_match() sets match_start */ + + if (s->match_length <= 5 && (s->strategy == Z_FILTERED +#if TOO_FAR <= 32767 + || (s->match_length == MIN_MATCH && + s->strstart - s->match_start > TOO_FAR) +#endif + )) { + + /* If prev_match is also MIN_MATCH, match_start is garbage + * but we will ignore the current match anyway. + */ + s->match_length = MIN_MATCH-1; + } + } + /* If there was a match at the previous step and the current + * match is not better, output the previous match: + */ + if (s->prev_length >= MIN_MATCH && s->match_length <= s->prev_length) { + uInt max_insert = s->strstart + s->lookahead - MIN_MATCH; + /* Do not insert strings in hash table beyond this. */ + + check_match(s, s->strstart-1, s->prev_match, s->prev_length); + + _tr_tally_dist(s, s->strstart -1 - s->prev_match, + s->prev_length - MIN_MATCH, bflush); + + /* Insert in hash table all strings up to the end of the match. + * strstart-1 and strstart are already inserted. If there is not + * enough lookahead, the last two strings are not inserted in + * the hash table. + */ + s->lookahead -= s->prev_length-1; + s->prev_length -= 2; + do { + if (++s->strstart <= max_insert) { + INSERT_STRING(s, s->strstart, hash_head); + } + } while (--s->prev_length != 0); + s->match_available = 0; + s->match_length = MIN_MATCH-1; + s->strstart++; + + if (bflush) FLUSH_BLOCK(s, 0); + + } else if (s->match_available) { + /* If there was no match at the previous position, output a + * single literal. If there was a match but the current match + * is longer, truncate the previous match to a single literal. + */ + Tracevv((stderr,"%c", s->window[s->strstart-1])); + _tr_tally_lit(s, s->window[s->strstart-1], bflush); + if (bflush) { + FLUSH_BLOCK_ONLY(s, 0); + } + s->strstart++; + s->lookahead--; + if (s->strm->avail_out == 0) return need_more; + } else { + /* There is no previous match to compare with, wait for + * the next step to decide. + */ + s->match_available = 1; + s->strstart++; + s->lookahead--; + } + } + Assert (flush != Z_NO_FLUSH, "no flush?"); + if (s->match_available) { + Tracevv((stderr,"%c", s->window[s->strstart-1])); + _tr_tally_lit(s, s->window[s->strstart-1], bflush); + s->match_available = 0; + } + FLUSH_BLOCK(s, flush == Z_FINISH); + return flush == Z_FINISH ? finish_done : block_done; +} +#endif /* FASTEST */ + +/* =========================================================================== + * For Z_RLE, simply look for runs of bytes, generate matches only of distance + * one. Do not maintain a hash table. (It will be regenerated if this run of + * deflate switches away from Z_RLE.) + */ +local block_state deflate_rle(s, flush) + deflate_state *s; + int flush; +{ + int bflush; /* set if current block must be flushed */ + uInt prev; /* byte at distance one to match */ + Bytef *scan, *strend; /* scan goes up to strend for length of run */ + + for (;;) { + /* Make sure that we always have enough lookahead, except + * at the end of the input file. We need MAX_MATCH bytes + * for the longest encodable run. + */ + if (s->lookahead < MAX_MATCH) { + fill_window(s); + if (s->lookahead < MAX_MATCH && flush == Z_NO_FLUSH) { + return need_more; + } + if (s->lookahead == 0) break; /* flush the current block */ + } + + /* See how many times the previous byte repeats */ + s->match_length = 0; + if (s->lookahead >= MIN_MATCH && s->strstart > 0) { + scan = s->window + s->strstart - 1; + prev = *scan; + if (prev == *++scan && prev == *++scan && prev == *++scan) { + strend = s->window + s->strstart + MAX_MATCH; + do { + } while (prev == *++scan && prev == *++scan && + prev == *++scan && prev == *++scan && + prev == *++scan && prev == *++scan && + prev == *++scan && prev == *++scan && + scan < strend); + s->match_length = MAX_MATCH - (int)(strend - scan); + if (s->match_length > s->lookahead) + s->match_length = s->lookahead; + } + } + + /* Emit match if have run of MIN_MATCH or longer, else emit literal */ + if (s->match_length >= MIN_MATCH) { + check_match(s, s->strstart, s->strstart - 1, s->match_length); + + _tr_tally_dist(s, 1, s->match_length - MIN_MATCH, bflush); + + s->lookahead -= s->match_length; + s->strstart += s->match_length; + s->match_length = 0; + } else { + /* No match, output a literal byte */ + Tracevv((stderr,"%c", s->window[s->strstart])); + _tr_tally_lit (s, s->window[s->strstart], bflush); + s->lookahead--; + s->strstart++; + } + if (bflush) FLUSH_BLOCK(s, 0); + } + FLUSH_BLOCK(s, flush == Z_FINISH); + return flush == Z_FINISH ? finish_done : block_done; +} + +/* =========================================================================== + * For Z_HUFFMAN_ONLY, do not look for matches. Do not maintain a hash table. + * (It will be regenerated if this run of deflate switches away from Huffman.) + */ +local block_state deflate_huff(s, flush) + deflate_state *s; + int flush; +{ + int bflush; /* set if current block must be flushed */ + + for (;;) { + /* Make sure that we have a literal to write. */ + if (s->lookahead == 0) { + fill_window(s); + if (s->lookahead == 0) { + if (flush == Z_NO_FLUSH) + return need_more; + break; /* flush the current block */ + } + } + + /* Output a literal byte */ + s->match_length = 0; + Tracevv((stderr,"%c", s->window[s->strstart])); + _tr_tally_lit (s, s->window[s->strstart], bflush); + s->lookahead--; + s->strstart++; + if (bflush) FLUSH_BLOCK(s, 0); + } + FLUSH_BLOCK(s, flush == Z_FINISH); + return flush == Z_FINISH ? finish_done : block_done; +} diff --git a/deps/zlib/deflate.h b/deps/zlib/deflate.h new file mode 100644 index 000000000..cbf0d1ea5 --- /dev/null +++ b/deps/zlib/deflate.h @@ -0,0 +1,342 @@ +/* deflate.h -- internal compression state + * Copyright (C) 1995-2010 Jean-loup Gailly + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* WARNING: this file should *not* be used by applications. It is + part of the implementation of the compression library and is + subject to change. Applications should only use zlib.h. + */ + +/* @(#) $Id$ */ + +#ifndef DEFLATE_H +#define DEFLATE_H + +#include "zutil.h" + +/* define NO_GZIP when compiling if you want to disable gzip header and + trailer creation by deflate(). NO_GZIP would be used to avoid linking in + the crc code when it is not needed. For shared libraries, gzip encoding + should be left enabled. */ +#ifndef NO_GZIP +# define GZIP +#endif + +/* =========================================================================== + * Internal compression state. + */ + +#define LENGTH_CODES 29 +/* number of length codes, not counting the special END_BLOCK code */ + +#define LITERALS 256 +/* number of literal bytes 0..255 */ + +#define L_CODES (LITERALS+1+LENGTH_CODES) +/* number of Literal or Length codes, including the END_BLOCK code */ + +#define D_CODES 30 +/* number of distance codes */ + +#define BL_CODES 19 +/* number of codes used to transfer the bit lengths */ + +#define HEAP_SIZE (2*L_CODES+1) +/* maximum heap size */ + +#define MAX_BITS 15 +/* All codes must not exceed MAX_BITS bits */ + +#define INIT_STATE 42 +#define EXTRA_STATE 69 +#define NAME_STATE 73 +#define COMMENT_STATE 91 +#define HCRC_STATE 103 +#define BUSY_STATE 113 +#define FINISH_STATE 666 +/* Stream status */ + + +/* Data structure describing a single value and its code string. */ +typedef struct ct_data_s { + union { + ush freq; /* frequency count */ + ush code; /* bit string */ + } fc; + union { + ush dad; /* father node in Huffman tree */ + ush len; /* length of bit string */ + } dl; +} FAR ct_data; + +#define Freq fc.freq +#define Code fc.code +#define Dad dl.dad +#define Len dl.len + +typedef struct static_tree_desc_s static_tree_desc; + +typedef struct tree_desc_s { + ct_data *dyn_tree; /* the dynamic tree */ + int max_code; /* largest code with non zero frequency */ + static_tree_desc *stat_desc; /* the corresponding static tree */ +} FAR tree_desc; + +typedef ush Pos; +typedef Pos FAR Posf; +typedef unsigned IPos; + +/* A Pos is an index in the character window. We use short instead of int to + * save space in the various tables. IPos is used only for parameter passing. + */ + +typedef struct internal_state { + z_streamp strm; /* pointer back to this zlib stream */ + int status; /* as the name implies */ + Bytef *pending_buf; /* output still pending */ + ulg pending_buf_size; /* size of pending_buf */ + Bytef *pending_out; /* next pending byte to output to the stream */ + uInt pending; /* nb of bytes in the pending buffer */ + int wrap; /* bit 0 true for zlib, bit 1 true for gzip */ + gz_headerp gzhead; /* gzip header information to write */ + uInt gzindex; /* where in extra, name, or comment */ + Byte method; /* STORED (for zip only) or DEFLATED */ + int last_flush; /* value of flush param for previous deflate call */ + + /* used by deflate.c: */ + + uInt w_size; /* LZ77 window size (32K by default) */ + uInt w_bits; /* log2(w_size) (8..16) */ + uInt w_mask; /* w_size - 1 */ + + Bytef *window; + /* Sliding window. Input bytes are read into the second half of the window, + * and move to the first half later to keep a dictionary of at least wSize + * bytes. With this organization, matches are limited to a distance of + * wSize-MAX_MATCH bytes, but this ensures that IO is always + * performed with a length multiple of the block size. Also, it limits + * the window size to 64K, which is quite useful on MSDOS. + * To do: use the user input buffer as sliding window. + */ + + ulg window_size; + /* Actual size of window: 2*wSize, except when the user input buffer + * is directly used as sliding window. + */ + + Posf *prev; + /* Link to older string with same hash index. To limit the size of this + * array to 64K, this link is maintained only for the last 32K strings. + * An index in this array is thus a window index modulo 32K. + */ + + Posf *head; /* Heads of the hash chains or NIL. */ + + uInt ins_h; /* hash index of string to be inserted */ + uInt hash_size; /* number of elements in hash table */ + uInt hash_bits; /* log2(hash_size) */ + uInt hash_mask; /* hash_size-1 */ + + uInt hash_shift; + /* Number of bits by which ins_h must be shifted at each input + * step. It must be such that after MIN_MATCH steps, the oldest + * byte no longer takes part in the hash key, that is: + * hash_shift * MIN_MATCH >= hash_bits + */ + + long block_start; + /* Window position at the beginning of the current output block. Gets + * negative when the window is moved backwards. + */ + + uInt match_length; /* length of best match */ + IPos prev_match; /* previous match */ + int match_available; /* set if previous match exists */ + uInt strstart; /* start of string to insert */ + uInt match_start; /* start of matching string */ + uInt lookahead; /* number of valid bytes ahead in window */ + + uInt prev_length; + /* Length of the best match at previous step. Matches not greater than this + * are discarded. This is used in the lazy match evaluation. + */ + + uInt max_chain_length; + /* To speed up deflation, hash chains are never searched beyond this + * length. A higher limit improves compression ratio but degrades the + * speed. + */ + + uInt max_lazy_match; + /* Attempt to find a better match only when the current match is strictly + * smaller than this value. This mechanism is used only for compression + * levels >= 4. + */ +# define max_insert_length max_lazy_match + /* Insert new strings in the hash table only if the match length is not + * greater than this length. This saves time but degrades compression. + * max_insert_length is used only for compression levels <= 3. + */ + + int level; /* compression level (1..9) */ + int strategy; /* favor or force Huffman coding*/ + + uInt good_match; + /* Use a faster search when the previous match is longer than this */ + + int nice_match; /* Stop searching when current match exceeds this */ + + /* used by trees.c: */ + /* Didn't use ct_data typedef below to supress compiler warning */ + struct ct_data_s dyn_ltree[HEAP_SIZE]; /* literal and length tree */ + struct ct_data_s dyn_dtree[2*D_CODES+1]; /* distance tree */ + struct ct_data_s bl_tree[2*BL_CODES+1]; /* Huffman tree for bit lengths */ + + struct tree_desc_s l_desc; /* desc. for literal tree */ + struct tree_desc_s d_desc; /* desc. for distance tree */ + struct tree_desc_s bl_desc; /* desc. for bit length tree */ + + ush bl_count[MAX_BITS+1]; + /* number of codes at each bit length for an optimal tree */ + + int heap[2*L_CODES+1]; /* heap used to build the Huffman trees */ + int heap_len; /* number of elements in the heap */ + int heap_max; /* element of largest frequency */ + /* The sons of heap[n] are heap[2*n] and heap[2*n+1]. heap[0] is not used. + * The same heap array is used to build all trees. + */ + + uch depth[2*L_CODES+1]; + /* Depth of each subtree used as tie breaker for trees of equal frequency + */ + + uchf *l_buf; /* buffer for literals or lengths */ + + uInt lit_bufsize; + /* Size of match buffer for literals/lengths. There are 4 reasons for + * limiting lit_bufsize to 64K: + * - frequencies can be kept in 16 bit counters + * - if compression is not successful for the first block, all input + * data is still in the window so we can still emit a stored block even + * when input comes from standard input. (This can also be done for + * all blocks if lit_bufsize is not greater than 32K.) + * - if compression is not successful for a file smaller than 64K, we can + * even emit a stored file instead of a stored block (saving 5 bytes). + * This is applicable only for zip (not gzip or zlib). + * - creating new Huffman trees less frequently may not provide fast + * adaptation to changes in the input data statistics. (Take for + * example a binary file with poorly compressible code followed by + * a highly compressible string table.) Smaller buffer sizes give + * fast adaptation but have of course the overhead of transmitting + * trees more frequently. + * - I can't count above 4 + */ + + uInt last_lit; /* running index in l_buf */ + + ushf *d_buf; + /* Buffer for distances. To simplify the code, d_buf and l_buf have + * the same number of elements. To use different lengths, an extra flag + * array would be necessary. + */ + + ulg opt_len; /* bit length of current block with optimal trees */ + ulg static_len; /* bit length of current block with static trees */ + uInt matches; /* number of string matches in current block */ + int last_eob_len; /* bit length of EOB code for last block */ + +#ifdef DEBUG + ulg compressed_len; /* total bit length of compressed file mod 2^32 */ + ulg bits_sent; /* bit length of compressed data sent mod 2^32 */ +#endif + + ush bi_buf; + /* Output buffer. bits are inserted starting at the bottom (least + * significant bits). + */ + int bi_valid; + /* Number of valid bits in bi_buf. All bits above the last valid bit + * are always zero. + */ + + ulg high_water; + /* High water mark offset in window for initialized bytes -- bytes above + * this are set to zero in order to avoid memory check warnings when + * longest match routines access bytes past the input. This is then + * updated to the new high water mark. + */ + +} FAR deflate_state; + +/* Output a byte on the stream. + * IN assertion: there is enough room in pending_buf. + */ +#define put_byte(s, c) {s->pending_buf[s->pending++] = (c);} + + +#define MIN_LOOKAHEAD (MAX_MATCH+MIN_MATCH+1) +/* Minimum amount of lookahead, except at the end of the input file. + * See deflate.c for comments about the MIN_MATCH+1. + */ + +#define MAX_DIST(s) ((s)->w_size-MIN_LOOKAHEAD) +/* In order to simplify the code, particularly on 16 bit machines, match + * distances are limited to MAX_DIST instead of WSIZE. + */ + +#define WIN_INIT MAX_MATCH +/* Number of bytes after end of data in window to initialize in order to avoid + memory checker errors from longest match routines */ + + /* in trees.c */ +void ZLIB_INTERNAL _tr_init OF((deflate_state *s)); +int ZLIB_INTERNAL _tr_tally OF((deflate_state *s, unsigned dist, unsigned lc)); +void ZLIB_INTERNAL _tr_flush_block OF((deflate_state *s, charf *buf, + ulg stored_len, int last)); +void ZLIB_INTERNAL _tr_align OF((deflate_state *s)); +void ZLIB_INTERNAL _tr_stored_block OF((deflate_state *s, charf *buf, + ulg stored_len, int last)); + +#define d_code(dist) \ + ((dist) < 256 ? _dist_code[dist] : _dist_code[256+((dist)>>7)]) +/* Mapping from a distance to a distance code. dist is the distance - 1 and + * must not have side effects. _dist_code[256] and _dist_code[257] are never + * used. + */ + +#ifndef DEBUG +/* Inline versions of _tr_tally for speed: */ + +#if defined(GEN_TREES_H) || !defined(STDC) + extern uch ZLIB_INTERNAL _length_code[]; + extern uch ZLIB_INTERNAL _dist_code[]; +#else + extern const uch ZLIB_INTERNAL _length_code[]; + extern const uch ZLIB_INTERNAL _dist_code[]; +#endif + +# define _tr_tally_lit(s, c, flush) \ + { uch cc = (c); \ + s->d_buf[s->last_lit] = 0; \ + s->l_buf[s->last_lit++] = cc; \ + s->dyn_ltree[cc].Freq++; \ + flush = (s->last_lit == s->lit_bufsize-1); \ + } +# define _tr_tally_dist(s, distance, length, flush) \ + { uch len = (length); \ + ush dist = (distance); \ + s->d_buf[s->last_lit] = dist; \ + s->l_buf[s->last_lit++] = len; \ + dist--; \ + s->dyn_ltree[_length_code[len]+LITERALS+1].Freq++; \ + s->dyn_dtree[d_code(dist)].Freq++; \ + flush = (s->last_lit == s->lit_bufsize-1); \ + } +#else +# define _tr_tally_lit(s, c, flush) flush = _tr_tally(s, 0, c) +# define _tr_tally_dist(s, distance, length, flush) \ + flush = _tr_tally(s, distance, length) +#endif + +#endif /* DEFLATE_H */ diff --git a/deps/zlib/inffast.c b/deps/zlib/inffast.c new file mode 100644 index 000000000..2f1d60b43 --- /dev/null +++ b/deps/zlib/inffast.c @@ -0,0 +1,340 @@ +/* inffast.c -- fast decoding + * Copyright (C) 1995-2008, 2010 Mark Adler + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +#include "zutil.h" +#include "inftrees.h" +#include "inflate.h" +#include "inffast.h" + +#ifndef ASMINF + +/* Allow machine dependent optimization for post-increment or pre-increment. + Based on testing to date, + Pre-increment preferred for: + - PowerPC G3 (Adler) + - MIPS R5000 (Randers-Pehrson) + Post-increment preferred for: + - none + No measurable difference: + - Pentium III (Anderson) + - M68060 (Nikl) + */ +#ifdef POSTINC +# define OFF 0 +# define PUP(a) *(a)++ +#else +# define OFF 1 +# define PUP(a) *++(a) +#endif + +/* + Decode literal, length, and distance codes and write out the resulting + literal and match bytes until either not enough input or output is + available, an end-of-block is encountered, or a data error is encountered. + When large enough input and output buffers are supplied to inflate(), for + example, a 16K input buffer and a 64K output buffer, more than 95% of the + inflate execution time is spent in this routine. + + Entry assumptions: + + state->mode == LEN + strm->avail_in >= 6 + strm->avail_out >= 258 + start >= strm->avail_out + state->bits < 8 + + On return, state->mode is one of: + + LEN -- ran out of enough output space or enough available input + TYPE -- reached end of block code, inflate() to interpret next block + BAD -- error in block data + + Notes: + + - The maximum input bits used by a length/distance pair is 15 bits for the + length code, 5 bits for the length extra, 15 bits for the distance code, + and 13 bits for the distance extra. This totals 48 bits, or six bytes. + Therefore if strm->avail_in >= 6, then there is enough input to avoid + checking for available input while decoding. + + - The maximum bytes that a single length/distance pair can output is 258 + bytes, which is the maximum length that can be coded. inflate_fast() + requires strm->avail_out >= 258 for each loop to avoid checking for + output space. + */ +void ZLIB_INTERNAL inflate_fast(strm, start) +z_streamp strm; +unsigned start; /* inflate()'s starting value for strm->avail_out */ +{ + struct inflate_state FAR *state; + unsigned char FAR *in; /* local strm->next_in */ + unsigned char FAR *last; /* while in < last, enough input available */ + unsigned char FAR *out; /* local strm->next_out */ + unsigned char FAR *beg; /* inflate()'s initial strm->next_out */ + unsigned char FAR *end; /* while out < end, enough space available */ +#ifdef INFLATE_STRICT + unsigned dmax; /* maximum distance from zlib header */ +#endif + unsigned wsize; /* window size or zero if not using window */ + unsigned whave; /* valid bytes in the window */ + unsigned wnext; /* window write index */ + unsigned char FAR *window; /* allocated sliding window, if wsize != 0 */ + unsigned long hold; /* local strm->hold */ + unsigned bits; /* local strm->bits */ + code const FAR *lcode; /* local strm->lencode */ + code const FAR *dcode; /* local strm->distcode */ + unsigned lmask; /* mask for first level of length codes */ + unsigned dmask; /* mask for first level of distance codes */ + code here; /* retrieved table entry */ + unsigned op; /* code bits, operation, extra bits, or */ + /* window position, window bytes to copy */ + unsigned len; /* match length, unused bytes */ + unsigned dist; /* match distance */ + unsigned char FAR *from; /* where to copy match from */ + + /* copy state to local variables */ + state = (struct inflate_state FAR *)strm->state; + in = strm->next_in - OFF; + last = in + (strm->avail_in - 5); + out = strm->next_out - OFF; + beg = out - (start - strm->avail_out); + end = out + (strm->avail_out - 257); +#ifdef INFLATE_STRICT + dmax = state->dmax; +#endif + wsize = state->wsize; + whave = state->whave; + wnext = state->wnext; + window = state->window; + hold = state->hold; + bits = state->bits; + lcode = state->lencode; + dcode = state->distcode; + lmask = (1U << state->lenbits) - 1; + dmask = (1U << state->distbits) - 1; + + /* decode literals and length/distances until end-of-block or not enough + input data or output space */ + do { + if (bits < 15) { + hold += (unsigned long)(PUP(in)) << bits; + bits += 8; + hold += (unsigned long)(PUP(in)) << bits; + bits += 8; + } + here = lcode[hold & lmask]; + dolen: + op = (unsigned)(here.bits); + hold >>= op; + bits -= op; + op = (unsigned)(here.op); + if (op == 0) { /* literal */ + Tracevv((stderr, here.val >= 0x20 && here.val < 0x7f ? + "inflate: literal '%c'\n" : + "inflate: literal 0x%02x\n", here.val)); + PUP(out) = (unsigned char)(here.val); + } + else if (op & 16) { /* length base */ + len = (unsigned)(here.val); + op &= 15; /* number of extra bits */ + if (op) { + if (bits < op) { + hold += (unsigned long)(PUP(in)) << bits; + bits += 8; + } + len += (unsigned)hold & ((1U << op) - 1); + hold >>= op; + bits -= op; + } + Tracevv((stderr, "inflate: length %u\n", len)); + if (bits < 15) { + hold += (unsigned long)(PUP(in)) << bits; + bits += 8; + hold += (unsigned long)(PUP(in)) << bits; + bits += 8; + } + here = dcode[hold & dmask]; + dodist: + op = (unsigned)(here.bits); + hold >>= op; + bits -= op; + op = (unsigned)(here.op); + if (op & 16) { /* distance base */ + dist = (unsigned)(here.val); + op &= 15; /* number of extra bits */ + if (bits < op) { + hold += (unsigned long)(PUP(in)) << bits; + bits += 8; + if (bits < op) { + hold += (unsigned long)(PUP(in)) << bits; + bits += 8; + } + } + dist += (unsigned)hold & ((1U << op) - 1); +#ifdef INFLATE_STRICT + if (dist > dmax) { + strm->msg = (char *)"invalid distance too far back"; + state->mode = BAD; + break; + } +#endif + hold >>= op; + bits -= op; + Tracevv((stderr, "inflate: distance %u\n", dist)); + op = (unsigned)(out - beg); /* max distance in output */ + if (dist > op) { /* see if copy from window */ + op = dist - op; /* distance back in window */ + if (op > whave) { + if (state->sane) { + strm->msg = + (char *)"invalid distance too far back"; + state->mode = BAD; + break; + } +#ifdef INFLATE_ALLOW_INVALID_DISTANCE_TOOFAR_ARRR + if (len <= op - whave) { + do { + PUP(out) = 0; + } while (--len); + continue; + } + len -= op - whave; + do { + PUP(out) = 0; + } while (--op > whave); + if (op == 0) { + from = out - dist; + do { + PUP(out) = PUP(from); + } while (--len); + continue; + } +#endif + } + from = window - OFF; + if (wnext == 0) { /* very common case */ + from += wsize - op; + if (op < len) { /* some from window */ + len -= op; + do { + PUP(out) = PUP(from); + } while (--op); + from = out - dist; /* rest from output */ + } + } + else if (wnext < op) { /* wrap around window */ + from += wsize + wnext - op; + op -= wnext; + if (op < len) { /* some from end of window */ + len -= op; + do { + PUP(out) = PUP(from); + } while (--op); + from = window - OFF; + if (wnext < len) { /* some from start of window */ + op = wnext; + len -= op; + do { + PUP(out) = PUP(from); + } while (--op); + from = out - dist; /* rest from output */ + } + } + } + else { /* contiguous in window */ + from += wnext - op; + if (op < len) { /* some from window */ + len -= op; + do { + PUP(out) = PUP(from); + } while (--op); + from = out - dist; /* rest from output */ + } + } + while (len > 2) { + PUP(out) = PUP(from); + PUP(out) = PUP(from); + PUP(out) = PUP(from); + len -= 3; + } + if (len) { + PUP(out) = PUP(from); + if (len > 1) + PUP(out) = PUP(from); + } + } + else { + from = out - dist; /* copy direct from output */ + do { /* minimum length is three */ + PUP(out) = PUP(from); + PUP(out) = PUP(from); + PUP(out) = PUP(from); + len -= 3; + } while (len > 2); + if (len) { + PUP(out) = PUP(from); + if (len > 1) + PUP(out) = PUP(from); + } + } + } + else if ((op & 64) == 0) { /* 2nd level distance code */ + here = dcode[here.val + (hold & ((1U << op) - 1))]; + goto dodist; + } + else { + strm->msg = (char *)"invalid distance code"; + state->mode = BAD; + break; + } + } + else if ((op & 64) == 0) { /* 2nd level length code */ + here = lcode[here.val + (hold & ((1U << op) - 1))]; + goto dolen; + } + else if (op & 32) { /* end-of-block */ + Tracevv((stderr, "inflate: end of block\n")); + state->mode = TYPE; + break; + } + else { + strm->msg = (char *)"invalid literal/length code"; + state->mode = BAD; + break; + } + } while (in < last && out < end); + + /* return unused bytes (on entry, bits < 8, so in won't go too far back) */ + len = bits >> 3; + in -= len; + bits -= len << 3; + hold &= (1U << bits) - 1; + + /* update state and return */ + strm->next_in = in + OFF; + strm->next_out = out + OFF; + strm->avail_in = (unsigned)(in < last ? 5 + (last - in) : 5 - (in - last)); + strm->avail_out = (unsigned)(out < end ? + 257 + (end - out) : 257 - (out - end)); + state->hold = hold; + state->bits = bits; + return; +} + +/* + inflate_fast() speedups that turned out slower (on a PowerPC G3 750CXe): + - Using bit fields for code structure + - Different op definition to avoid & for extra bits (do & for table bits) + - Three separate decoding do-loops for direct, window, and wnext == 0 + - Special case for distance > 1 copies to do overlapped load and store copy + - Explicit branch predictions (based on measured branch probabilities) + - Deferring match copy and interspersed it with decoding subsequent codes + - Swapping literal/length else + - Swapping window/direct else + - Larger unrolled copy loops (three is about right) + - Moving len -= 3 statement into middle of loop + */ + +#endif /* !ASMINF */ diff --git a/deps/zlib/inffast.h b/deps/zlib/inffast.h new file mode 100644 index 000000000..e5c1aa4ca --- /dev/null +++ b/deps/zlib/inffast.h @@ -0,0 +1,11 @@ +/* inffast.h -- header to use inffast.c + * Copyright (C) 1995-2003, 2010 Mark Adler + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* WARNING: this file should *not* be used by applications. It is + part of the implementation of the compression library and is + subject to change. Applications should only use zlib.h. + */ + +void ZLIB_INTERNAL inflate_fast OF((z_streamp strm, unsigned start)); diff --git a/deps/zlib/inffixed.h b/deps/zlib/inffixed.h new file mode 100644 index 000000000..75ed4b597 --- /dev/null +++ b/deps/zlib/inffixed.h @@ -0,0 +1,94 @@ + /* inffixed.h -- table for decoding fixed codes + * Generated automatically by makefixed(). + */ + + /* WARNING: this file should *not* be used by applications. It + is part of the implementation of the compression library and + is subject to change. Applications should only use zlib.h. + */ + + static const code lenfix[512] = { + {96,7,0},{0,8,80},{0,8,16},{20,8,115},{18,7,31},{0,8,112},{0,8,48}, + {0,9,192},{16,7,10},{0,8,96},{0,8,32},{0,9,160},{0,8,0},{0,8,128}, + {0,8,64},{0,9,224},{16,7,6},{0,8,88},{0,8,24},{0,9,144},{19,7,59}, + {0,8,120},{0,8,56},{0,9,208},{17,7,17},{0,8,104},{0,8,40},{0,9,176}, + {0,8,8},{0,8,136},{0,8,72},{0,9,240},{16,7,4},{0,8,84},{0,8,20}, + {21,8,227},{19,7,43},{0,8,116},{0,8,52},{0,9,200},{17,7,13},{0,8,100}, + {0,8,36},{0,9,168},{0,8,4},{0,8,132},{0,8,68},{0,9,232},{16,7,8}, + {0,8,92},{0,8,28},{0,9,152},{20,7,83},{0,8,124},{0,8,60},{0,9,216}, + {18,7,23},{0,8,108},{0,8,44},{0,9,184},{0,8,12},{0,8,140},{0,8,76}, + {0,9,248},{16,7,3},{0,8,82},{0,8,18},{21,8,163},{19,7,35},{0,8,114}, + {0,8,50},{0,9,196},{17,7,11},{0,8,98},{0,8,34},{0,9,164},{0,8,2}, + {0,8,130},{0,8,66},{0,9,228},{16,7,7},{0,8,90},{0,8,26},{0,9,148}, + {20,7,67},{0,8,122},{0,8,58},{0,9,212},{18,7,19},{0,8,106},{0,8,42}, + {0,9,180},{0,8,10},{0,8,138},{0,8,74},{0,9,244},{16,7,5},{0,8,86}, + {0,8,22},{64,8,0},{19,7,51},{0,8,118},{0,8,54},{0,9,204},{17,7,15}, + {0,8,102},{0,8,38},{0,9,172},{0,8,6},{0,8,134},{0,8,70},{0,9,236}, + {16,7,9},{0,8,94},{0,8,30},{0,9,156},{20,7,99},{0,8,126},{0,8,62}, + {0,9,220},{18,7,27},{0,8,110},{0,8,46},{0,9,188},{0,8,14},{0,8,142}, + {0,8,78},{0,9,252},{96,7,0},{0,8,81},{0,8,17},{21,8,131},{18,7,31}, + {0,8,113},{0,8,49},{0,9,194},{16,7,10},{0,8,97},{0,8,33},{0,9,162}, + {0,8,1},{0,8,129},{0,8,65},{0,9,226},{16,7,6},{0,8,89},{0,8,25}, + {0,9,146},{19,7,59},{0,8,121},{0,8,57},{0,9,210},{17,7,17},{0,8,105}, + {0,8,41},{0,9,178},{0,8,9},{0,8,137},{0,8,73},{0,9,242},{16,7,4}, + {0,8,85},{0,8,21},{16,8,258},{19,7,43},{0,8,117},{0,8,53},{0,9,202}, + {17,7,13},{0,8,101},{0,8,37},{0,9,170},{0,8,5},{0,8,133},{0,8,69}, + {0,9,234},{16,7,8},{0,8,93},{0,8,29},{0,9,154},{20,7,83},{0,8,125}, + {0,8,61},{0,9,218},{18,7,23},{0,8,109},{0,8,45},{0,9,186},{0,8,13}, + {0,8,141},{0,8,77},{0,9,250},{16,7,3},{0,8,83},{0,8,19},{21,8,195}, + {19,7,35},{0,8,115},{0,8,51},{0,9,198},{17,7,11},{0,8,99},{0,8,35}, + {0,9,166},{0,8,3},{0,8,131},{0,8,67},{0,9,230},{16,7,7},{0,8,91}, + {0,8,27},{0,9,150},{20,7,67},{0,8,123},{0,8,59},{0,9,214},{18,7,19}, + {0,8,107},{0,8,43},{0,9,182},{0,8,11},{0,8,139},{0,8,75},{0,9,246}, + {16,7,5},{0,8,87},{0,8,23},{64,8,0},{19,7,51},{0,8,119},{0,8,55}, + {0,9,206},{17,7,15},{0,8,103},{0,8,39},{0,9,174},{0,8,7},{0,8,135}, + {0,8,71},{0,9,238},{16,7,9},{0,8,95},{0,8,31},{0,9,158},{20,7,99}, + {0,8,127},{0,8,63},{0,9,222},{18,7,27},{0,8,111},{0,8,47},{0,9,190}, + {0,8,15},{0,8,143},{0,8,79},{0,9,254},{96,7,0},{0,8,80},{0,8,16}, + {20,8,115},{18,7,31},{0,8,112},{0,8,48},{0,9,193},{16,7,10},{0,8,96}, + {0,8,32},{0,9,161},{0,8,0},{0,8,128},{0,8,64},{0,9,225},{16,7,6}, + {0,8,88},{0,8,24},{0,9,145},{19,7,59},{0,8,120},{0,8,56},{0,9,209}, + {17,7,17},{0,8,104},{0,8,40},{0,9,177},{0,8,8},{0,8,136},{0,8,72}, + {0,9,241},{16,7,4},{0,8,84},{0,8,20},{21,8,227},{19,7,43},{0,8,116}, + {0,8,52},{0,9,201},{17,7,13},{0,8,100},{0,8,36},{0,9,169},{0,8,4}, + {0,8,132},{0,8,68},{0,9,233},{16,7,8},{0,8,92},{0,8,28},{0,9,153}, + {20,7,83},{0,8,124},{0,8,60},{0,9,217},{18,7,23},{0,8,108},{0,8,44}, + {0,9,185},{0,8,12},{0,8,140},{0,8,76},{0,9,249},{16,7,3},{0,8,82}, + {0,8,18},{21,8,163},{19,7,35},{0,8,114},{0,8,50},{0,9,197},{17,7,11}, + {0,8,98},{0,8,34},{0,9,165},{0,8,2},{0,8,130},{0,8,66},{0,9,229}, + {16,7,7},{0,8,90},{0,8,26},{0,9,149},{20,7,67},{0,8,122},{0,8,58}, + {0,9,213},{18,7,19},{0,8,106},{0,8,42},{0,9,181},{0,8,10},{0,8,138}, + {0,8,74},{0,9,245},{16,7,5},{0,8,86},{0,8,22},{64,8,0},{19,7,51}, + {0,8,118},{0,8,54},{0,9,205},{17,7,15},{0,8,102},{0,8,38},{0,9,173}, + {0,8,6},{0,8,134},{0,8,70},{0,9,237},{16,7,9},{0,8,94},{0,8,30}, + {0,9,157},{20,7,99},{0,8,126},{0,8,62},{0,9,221},{18,7,27},{0,8,110}, + {0,8,46},{0,9,189},{0,8,14},{0,8,142},{0,8,78},{0,9,253},{96,7,0}, + {0,8,81},{0,8,17},{21,8,131},{18,7,31},{0,8,113},{0,8,49},{0,9,195}, + {16,7,10},{0,8,97},{0,8,33},{0,9,163},{0,8,1},{0,8,129},{0,8,65}, + {0,9,227},{16,7,6},{0,8,89},{0,8,25},{0,9,147},{19,7,59},{0,8,121}, + {0,8,57},{0,9,211},{17,7,17},{0,8,105},{0,8,41},{0,9,179},{0,8,9}, + {0,8,137},{0,8,73},{0,9,243},{16,7,4},{0,8,85},{0,8,21},{16,8,258}, + {19,7,43},{0,8,117},{0,8,53},{0,9,203},{17,7,13},{0,8,101},{0,8,37}, + {0,9,171},{0,8,5},{0,8,133},{0,8,69},{0,9,235},{16,7,8},{0,8,93}, + {0,8,29},{0,9,155},{20,7,83},{0,8,125},{0,8,61},{0,9,219},{18,7,23}, + {0,8,109},{0,8,45},{0,9,187},{0,8,13},{0,8,141},{0,8,77},{0,9,251}, + {16,7,3},{0,8,83},{0,8,19},{21,8,195},{19,7,35},{0,8,115},{0,8,51}, + {0,9,199},{17,7,11},{0,8,99},{0,8,35},{0,9,167},{0,8,3},{0,8,131}, + {0,8,67},{0,9,231},{16,7,7},{0,8,91},{0,8,27},{0,9,151},{20,7,67}, + {0,8,123},{0,8,59},{0,9,215},{18,7,19},{0,8,107},{0,8,43},{0,9,183}, + {0,8,11},{0,8,139},{0,8,75},{0,9,247},{16,7,5},{0,8,87},{0,8,23}, + {64,8,0},{19,7,51},{0,8,119},{0,8,55},{0,9,207},{17,7,15},{0,8,103}, + {0,8,39},{0,9,175},{0,8,7},{0,8,135},{0,8,71},{0,9,239},{16,7,9}, + {0,8,95},{0,8,31},{0,9,159},{20,7,99},{0,8,127},{0,8,63},{0,9,223}, + {18,7,27},{0,8,111},{0,8,47},{0,9,191},{0,8,15},{0,8,143},{0,8,79}, + {0,9,255} + }; + + static const code distfix[32] = { + {16,5,1},{23,5,257},{19,5,17},{27,5,4097},{17,5,5},{25,5,1025}, + {21,5,65},{29,5,16385},{16,5,3},{24,5,513},{20,5,33},{28,5,8193}, + {18,5,9},{26,5,2049},{22,5,129},{64,5,0},{16,5,2},{23,5,385}, + {19,5,25},{27,5,6145},{17,5,7},{25,5,1537},{21,5,97},{29,5,24577}, + {16,5,4},{24,5,769},{20,5,49},{28,5,12289},{18,5,13},{26,5,3073}, + {22,5,193},{64,5,0} + }; diff --git a/deps/zlib/inflate.c b/deps/zlib/inflate.c new file mode 100644 index 000000000..a8431abea --- /dev/null +++ b/deps/zlib/inflate.c @@ -0,0 +1,1480 @@ +/* inflate.c -- zlib decompression + * Copyright (C) 1995-2010 Mark Adler + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* + * Change history: + * + * 1.2.beta0 24 Nov 2002 + * - First version -- complete rewrite of inflate to simplify code, avoid + * creation of window when not needed, minimize use of window when it is + * needed, make inffast.c even faster, implement gzip decoding, and to + * improve code readability and style over the previous zlib inflate code + * + * 1.2.beta1 25 Nov 2002 + * - Use pointers for available input and output checking in inffast.c + * - Remove input and output counters in inffast.c + * - Change inffast.c entry and loop from avail_in >= 7 to >= 6 + * - Remove unnecessary second byte pull from length extra in inffast.c + * - Unroll direct copy to three copies per loop in inffast.c + * + * 1.2.beta2 4 Dec 2002 + * - Change external routine names to reduce potential conflicts + * - Correct filename to inffixed.h for fixed tables in inflate.c + * - Make hbuf[] unsigned char to match parameter type in inflate.c + * - Change strm->next_out[-state->offset] to *(strm->next_out - state->offset) + * to avoid negation problem on Alphas (64 bit) in inflate.c + * + * 1.2.beta3 22 Dec 2002 + * - Add comments on state->bits assertion in inffast.c + * - Add comments on op field in inftrees.h + * - Fix bug in reuse of allocated window after inflateReset() + * - Remove bit fields--back to byte structure for speed + * - Remove distance extra == 0 check in inflate_fast()--only helps for lengths + * - Change post-increments to pre-increments in inflate_fast(), PPC biased? + * - Add compile time option, POSTINC, to use post-increments instead (Intel?) + * - Make MATCH copy in inflate() much faster for when inflate_fast() not used + * - Use local copies of stream next and avail values, as well as local bit + * buffer and bit count in inflate()--for speed when inflate_fast() not used + * + * 1.2.beta4 1 Jan 2003 + * - Split ptr - 257 statements in inflate_table() to avoid compiler warnings + * - Move a comment on output buffer sizes from inffast.c to inflate.c + * - Add comments in inffast.c to introduce the inflate_fast() routine + * - Rearrange window copies in inflate_fast() for speed and simplification + * - Unroll last copy for window match in inflate_fast() + * - Use local copies of window variables in inflate_fast() for speed + * - Pull out common wnext == 0 case for speed in inflate_fast() + * - Make op and len in inflate_fast() unsigned for consistency + * - Add FAR to lcode and dcode declarations in inflate_fast() + * - Simplified bad distance check in inflate_fast() + * - Added inflateBackInit(), inflateBack(), and inflateBackEnd() in new + * source file infback.c to provide a call-back interface to inflate for + * programs like gzip and unzip -- uses window as output buffer to avoid + * window copying + * + * 1.2.beta5 1 Jan 2003 + * - Improved inflateBack() interface to allow the caller to provide initial + * input in strm. + * - Fixed stored blocks bug in inflateBack() + * + * 1.2.beta6 4 Jan 2003 + * - Added comments in inffast.c on effectiveness of POSTINC + * - Typecasting all around to reduce compiler warnings + * - Changed loops from while (1) or do {} while (1) to for (;;), again to + * make compilers happy + * - Changed type of window in inflateBackInit() to unsigned char * + * + * 1.2.beta7 27 Jan 2003 + * - Changed many types to unsigned or unsigned short to avoid warnings + * - Added inflateCopy() function + * + * 1.2.0 9 Mar 2003 + * - Changed inflateBack() interface to provide separate opaque descriptors + * for the in() and out() functions + * - Changed inflateBack() argument and in_func typedef to swap the length + * and buffer address return values for the input function + * - Check next_in and next_out for Z_NULL on entry to inflate() + * + * The history for versions after 1.2.0 are in ChangeLog in zlib distribution. + */ + +#include "zutil.h" +#include "inftrees.h" +#include "inflate.h" +#include "inffast.h" + +#ifdef MAKEFIXED +# ifndef BUILDFIXED +# define BUILDFIXED +# endif +#endif + +/* function prototypes */ +local void fixedtables OF((struct inflate_state FAR *state)); +local int updatewindow OF((z_streamp strm, unsigned out)); +#ifdef BUILDFIXED + void makefixed OF((void)); +#endif +local unsigned syncsearch OF((unsigned FAR *have, unsigned char FAR *buf, + unsigned len)); + +int ZEXPORT inflateReset(strm) +z_streamp strm; +{ + struct inflate_state FAR *state; + + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + state = (struct inflate_state FAR *)strm->state; + strm->total_in = strm->total_out = state->total = 0; + strm->msg = Z_NULL; + strm->adler = 1; /* to support ill-conceived Java test suite */ + state->mode = HEAD; + state->last = 0; + state->havedict = 0; + state->dmax = 32768U; + state->head = Z_NULL; + state->wsize = 0; + state->whave = 0; + state->wnext = 0; + state->hold = 0; + state->bits = 0; + state->lencode = state->distcode = state->next = state->codes; + state->sane = 1; + state->back = -1; + Tracev((stderr, "inflate: reset\n")); + return Z_OK; +} + +int ZEXPORT inflateReset2(strm, windowBits) +z_streamp strm; +int windowBits; +{ + int wrap; + struct inflate_state FAR *state; + + /* get the state */ + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + state = (struct inflate_state FAR *)strm->state; + + /* extract wrap request from windowBits parameter */ + if (windowBits < 0) { + wrap = 0; + windowBits = -windowBits; + } + else { + wrap = (windowBits >> 4) + 1; +#ifdef GUNZIP + if (windowBits < 48) + windowBits &= 15; +#endif + } + + /* set number of window bits, free window if different */ + if (windowBits && (windowBits < 8 || windowBits > 15)) + return Z_STREAM_ERROR; + if (state->window != Z_NULL && state->wbits != (unsigned)windowBits) { + ZFREE(strm, state->window); + state->window = Z_NULL; + } + + /* update state and reset the rest of it */ + state->wrap = wrap; + state->wbits = (unsigned)windowBits; + return inflateReset(strm); +} + +int ZEXPORT inflateInit2_(strm, windowBits, version, stream_size) +z_streamp strm; +int windowBits; +const char *version; +int stream_size; +{ + int ret; + struct inflate_state FAR *state; + + if (version == Z_NULL || version[0] != ZLIB_VERSION[0] || + stream_size != (int)(sizeof(z_stream))) + return Z_VERSION_ERROR; + if (strm == Z_NULL) return Z_STREAM_ERROR; + strm->msg = Z_NULL; /* in case we return an error */ + if (strm->zalloc == (alloc_func)0) { + strm->zalloc = zcalloc; + strm->opaque = (voidpf)0; + } + if (strm->zfree == (free_func)0) strm->zfree = zcfree; + state = (struct inflate_state FAR *) + ZALLOC(strm, 1, sizeof(struct inflate_state)); + if (state == Z_NULL) return Z_MEM_ERROR; + Tracev((stderr, "inflate: allocated\n")); + strm->state = (struct internal_state FAR *)state; + state->window = Z_NULL; + ret = inflateReset2(strm, windowBits); + if (ret != Z_OK) { + ZFREE(strm, state); + strm->state = Z_NULL; + } + return ret; +} + +int ZEXPORT inflateInit_(strm, version, stream_size) +z_streamp strm; +const char *version; +int stream_size; +{ + return inflateInit2_(strm, DEF_WBITS, version, stream_size); +} + +int ZEXPORT inflatePrime(strm, bits, value) +z_streamp strm; +int bits; +int value; +{ + struct inflate_state FAR *state; + + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + state = (struct inflate_state FAR *)strm->state; + if (bits < 0) { + state->hold = 0; + state->bits = 0; + return Z_OK; + } + if (bits > 16 || state->bits + bits > 32) return Z_STREAM_ERROR; + value &= (1L << bits) - 1; + state->hold += value << state->bits; + state->bits += bits; + return Z_OK; +} + +/* + Return state with length and distance decoding tables and index sizes set to + fixed code decoding. Normally this returns fixed tables from inffixed.h. + If BUILDFIXED is defined, then instead this routine builds the tables the + first time it's called, and returns those tables the first time and + thereafter. This reduces the size of the code by about 2K bytes, in + exchange for a little execution time. However, BUILDFIXED should not be + used for threaded applications, since the rewriting of the tables and virgin + may not be thread-safe. + */ +local void fixedtables(state) +struct inflate_state FAR *state; +{ +#ifdef BUILDFIXED + static int virgin = 1; + static code *lenfix, *distfix; + static code fixed[544]; + + /* build fixed huffman tables if first call (may not be thread safe) */ + if (virgin) { + unsigned sym, bits; + static code *next; + + /* literal/length table */ + sym = 0; + while (sym < 144) state->lens[sym++] = 8; + while (sym < 256) state->lens[sym++] = 9; + while (sym < 280) state->lens[sym++] = 7; + while (sym < 288) state->lens[sym++] = 8; + next = fixed; + lenfix = next; + bits = 9; + inflate_table(LENS, state->lens, 288, &(next), &(bits), state->work); + + /* distance table */ + sym = 0; + while (sym < 32) state->lens[sym++] = 5; + distfix = next; + bits = 5; + inflate_table(DISTS, state->lens, 32, &(next), &(bits), state->work); + + /* do this just once */ + virgin = 0; + } +#else /* !BUILDFIXED */ +# include "inffixed.h" +#endif /* BUILDFIXED */ + state->lencode = lenfix; + state->lenbits = 9; + state->distcode = distfix; + state->distbits = 5; +} + +#ifdef MAKEFIXED +#include + +/* + Write out the inffixed.h that is #include'd above. Defining MAKEFIXED also + defines BUILDFIXED, so the tables are built on the fly. makefixed() writes + those tables to stdout, which would be piped to inffixed.h. A small program + can simply call makefixed to do this: + + void makefixed(void); + + int main(void) + { + makefixed(); + return 0; + } + + Then that can be linked with zlib built with MAKEFIXED defined and run: + + a.out > inffixed.h + */ +void makefixed() +{ + unsigned low, size; + struct inflate_state state; + + fixedtables(&state); + puts(" /* inffixed.h -- table for decoding fixed codes"); + puts(" * Generated automatically by makefixed()."); + puts(" */"); + puts(""); + puts(" /* WARNING: this file should *not* be used by applications."); + puts(" It is part of the implementation of this library and is"); + puts(" subject to change. Applications should only use zlib.h."); + puts(" */"); + puts(""); + size = 1U << 9; + printf(" static const code lenfix[%u] = {", size); + low = 0; + for (;;) { + if ((low % 7) == 0) printf("\n "); + printf("{%u,%u,%d}", state.lencode[low].op, state.lencode[low].bits, + state.lencode[low].val); + if (++low == size) break; + putchar(','); + } + puts("\n };"); + size = 1U << 5; + printf("\n static const code distfix[%u] = {", size); + low = 0; + for (;;) { + if ((low % 6) == 0) printf("\n "); + printf("{%u,%u,%d}", state.distcode[low].op, state.distcode[low].bits, + state.distcode[low].val); + if (++low == size) break; + putchar(','); + } + puts("\n };"); +} +#endif /* MAKEFIXED */ + +/* + Update the window with the last wsize (normally 32K) bytes written before + returning. If window does not exist yet, create it. This is only called + when a window is already in use, or when output has been written during this + inflate call, but the end of the deflate stream has not been reached yet. + It is also called to create a window for dictionary data when a dictionary + is loaded. + + Providing output buffers larger than 32K to inflate() should provide a speed + advantage, since only the last 32K of output is copied to the sliding window + upon return from inflate(), and since all distances after the first 32K of + output will fall in the output data, making match copies simpler and faster. + The advantage may be dependent on the size of the processor's data caches. + */ +local int updatewindow(strm, out) +z_streamp strm; +unsigned out; +{ + struct inflate_state FAR *state; + unsigned copy, dist; + + state = (struct inflate_state FAR *)strm->state; + + /* if it hasn't been done already, allocate space for the window */ + if (state->window == Z_NULL) { + state->window = (unsigned char FAR *) + ZALLOC(strm, 1U << state->wbits, + sizeof(unsigned char)); + if (state->window == Z_NULL) return 1; + } + + /* if window not in use yet, initialize */ + if (state->wsize == 0) { + state->wsize = 1U << state->wbits; + state->wnext = 0; + state->whave = 0; + } + + /* copy state->wsize or less output bytes into the circular window */ + copy = out - strm->avail_out; + if (copy >= state->wsize) { + zmemcpy(state->window, strm->next_out - state->wsize, state->wsize); + state->wnext = 0; + state->whave = state->wsize; + } + else { + dist = state->wsize - state->wnext; + if (dist > copy) dist = copy; + zmemcpy(state->window + state->wnext, strm->next_out - copy, dist); + copy -= dist; + if (copy) { + zmemcpy(state->window, strm->next_out - copy, copy); + state->wnext = copy; + state->whave = state->wsize; + } + else { + state->wnext += dist; + if (state->wnext == state->wsize) state->wnext = 0; + if (state->whave < state->wsize) state->whave += dist; + } + } + return 0; +} + +/* Macros for inflate(): */ + +/* check function to use adler32() for zlib or crc32() for gzip */ +#ifdef GUNZIP +# define UPDATE(check, buf, len) \ + (state->flags ? crc32(check, buf, len) : adler32(check, buf, len)) +#else +# define UPDATE(check, buf, len) adler32(check, buf, len) +#endif + +/* check macros for header crc */ +#ifdef GUNZIP +# define CRC2(check, word) \ + do { \ + hbuf[0] = (unsigned char)(word); \ + hbuf[1] = (unsigned char)((word) >> 8); \ + check = crc32(check, hbuf, 2); \ + } while (0) + +# define CRC4(check, word) \ + do { \ + hbuf[0] = (unsigned char)(word); \ + hbuf[1] = (unsigned char)((word) >> 8); \ + hbuf[2] = (unsigned char)((word) >> 16); \ + hbuf[3] = (unsigned char)((word) >> 24); \ + check = crc32(check, hbuf, 4); \ + } while (0) +#endif + +/* Load registers with state in inflate() for speed */ +#define LOAD() \ + do { \ + put = strm->next_out; \ + left = strm->avail_out; \ + next = strm->next_in; \ + have = strm->avail_in; \ + hold = state->hold; \ + bits = state->bits; \ + } while (0) + +/* Restore state from registers in inflate() */ +#define RESTORE() \ + do { \ + strm->next_out = put; \ + strm->avail_out = left; \ + strm->next_in = next; \ + strm->avail_in = have; \ + state->hold = hold; \ + state->bits = bits; \ + } while (0) + +/* Clear the input bit accumulator */ +#define INITBITS() \ + do { \ + hold = 0; \ + bits = 0; \ + } while (0) + +/* Get a byte of input into the bit accumulator, or return from inflate() + if there is no input available. */ +#define PULLBYTE() \ + do { \ + if (have == 0) goto inf_leave; \ + have--; \ + hold += (unsigned long)(*next++) << bits; \ + bits += 8; \ + } while (0) + +/* Assure that there are at least n bits in the bit accumulator. If there is + not enough available input to do that, then return from inflate(). */ +#define NEEDBITS(n) \ + do { \ + while (bits < (unsigned)(n)) \ + PULLBYTE(); \ + } while (0) + +/* Return the low n bits of the bit accumulator (n < 16) */ +#define BITS(n) \ + ((unsigned)hold & ((1U << (n)) - 1)) + +/* Remove n bits from the bit accumulator */ +#define DROPBITS(n) \ + do { \ + hold >>= (n); \ + bits -= (unsigned)(n); \ + } while (0) + +/* Remove zero to seven bits as needed to go to a byte boundary */ +#define BYTEBITS() \ + do { \ + hold >>= bits & 7; \ + bits -= bits & 7; \ + } while (0) + +/* Reverse the bytes in a 32-bit value */ +#define REVERSE(q) \ + ((((q) >> 24) & 0xff) + (((q) >> 8) & 0xff00) + \ + (((q) & 0xff00) << 8) + (((q) & 0xff) << 24)) + +/* + inflate() uses a state machine to process as much input data and generate as + much output data as possible before returning. The state machine is + structured roughly as follows: + + for (;;) switch (state) { + ... + case STATEn: + if (not enough input data or output space to make progress) + return; + ... make progress ... + state = STATEm; + break; + ... + } + + so when inflate() is called again, the same case is attempted again, and + if the appropriate resources are provided, the machine proceeds to the + next state. The NEEDBITS() macro is usually the way the state evaluates + whether it can proceed or should return. NEEDBITS() does the return if + the requested bits are not available. The typical use of the BITS macros + is: + + NEEDBITS(n); + ... do something with BITS(n) ... + DROPBITS(n); + + where NEEDBITS(n) either returns from inflate() if there isn't enough + input left to load n bits into the accumulator, or it continues. BITS(n) + gives the low n bits in the accumulator. When done, DROPBITS(n) drops + the low n bits off the accumulator. INITBITS() clears the accumulator + and sets the number of available bits to zero. BYTEBITS() discards just + enough bits to put the accumulator on a byte boundary. After BYTEBITS() + and a NEEDBITS(8), then BITS(8) would return the next byte in the stream. + + NEEDBITS(n) uses PULLBYTE() to get an available byte of input, or to return + if there is no input available. The decoding of variable length codes uses + PULLBYTE() directly in order to pull just enough bytes to decode the next + code, and no more. + + Some states loop until they get enough input, making sure that enough + state information is maintained to continue the loop where it left off + if NEEDBITS() returns in the loop. For example, want, need, and keep + would all have to actually be part of the saved state in case NEEDBITS() + returns: + + case STATEw: + while (want < need) { + NEEDBITS(n); + keep[want++] = BITS(n); + DROPBITS(n); + } + state = STATEx; + case STATEx: + + As shown above, if the next state is also the next case, then the break + is omitted. + + A state may also return if there is not enough output space available to + complete that state. Those states are copying stored data, writing a + literal byte, and copying a matching string. + + When returning, a "goto inf_leave" is used to update the total counters, + update the check value, and determine whether any progress has been made + during that inflate() call in order to return the proper return code. + Progress is defined as a change in either strm->avail_in or strm->avail_out. + When there is a window, goto inf_leave will update the window with the last + output written. If a goto inf_leave occurs in the middle of decompression + and there is no window currently, goto inf_leave will create one and copy + output to the window for the next call of inflate(). + + In this implementation, the flush parameter of inflate() only affects the + return code (per zlib.h). inflate() always writes as much as possible to + strm->next_out, given the space available and the provided input--the effect + documented in zlib.h of Z_SYNC_FLUSH. Furthermore, inflate() always defers + the allocation of and copying into a sliding window until necessary, which + provides the effect documented in zlib.h for Z_FINISH when the entire input + stream available. So the only thing the flush parameter actually does is: + when flush is set to Z_FINISH, inflate() cannot return Z_OK. Instead it + will return Z_BUF_ERROR if it has not reached the end of the stream. + */ + +int ZEXPORT inflate(strm, flush) +z_streamp strm; +int flush; +{ + struct inflate_state FAR *state; + unsigned char FAR *next; /* next input */ + unsigned char FAR *put; /* next output */ + unsigned have, left; /* available input and output */ + unsigned long hold; /* bit buffer */ + unsigned bits; /* bits in bit buffer */ + unsigned in, out; /* save starting available input and output */ + unsigned copy; /* number of stored or match bytes to copy */ + unsigned char FAR *from; /* where to copy match bytes from */ + code here; /* current decoding table entry */ + code last; /* parent table entry */ + unsigned len; /* length to copy for repeats, bits to drop */ + int ret; /* return code */ +#ifdef GUNZIP + unsigned char hbuf[4]; /* buffer for gzip header crc calculation */ +#endif + static const unsigned short order[19] = /* permutation of code lengths */ + {16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15}; + + if (strm == Z_NULL || strm->state == Z_NULL || strm->next_out == Z_NULL || + (strm->next_in == Z_NULL && strm->avail_in != 0)) + return Z_STREAM_ERROR; + + state = (struct inflate_state FAR *)strm->state; + if (state->mode == TYPE) state->mode = TYPEDO; /* skip check */ + LOAD(); + in = have; + out = left; + ret = Z_OK; + for (;;) + switch (state->mode) { + case HEAD: + if (state->wrap == 0) { + state->mode = TYPEDO; + break; + } + NEEDBITS(16); +#ifdef GUNZIP + if ((state->wrap & 2) && hold == 0x8b1f) { /* gzip header */ + state->check = crc32(0L, Z_NULL, 0); + CRC2(state->check, hold); + INITBITS(); + state->mode = FLAGS; + break; + } + state->flags = 0; /* expect zlib header */ + if (state->head != Z_NULL) + state->head->done = -1; + if (!(state->wrap & 1) || /* check if zlib header allowed */ +#else + if ( +#endif + ((BITS(8) << 8) + (hold >> 8)) % 31) { + strm->msg = (char *)"incorrect header check"; + state->mode = BAD; + break; + } + if (BITS(4) != Z_DEFLATED) { + strm->msg = (char *)"unknown compression method"; + state->mode = BAD; + break; + } + DROPBITS(4); + len = BITS(4) + 8; + if (state->wbits == 0) + state->wbits = len; + else if (len > state->wbits) { + strm->msg = (char *)"invalid window size"; + state->mode = BAD; + break; + } + state->dmax = 1U << len; + Tracev((stderr, "inflate: zlib header ok\n")); + strm->adler = state->check = adler32(0L, Z_NULL, 0); + state->mode = hold & 0x200 ? DICTID : TYPE; + INITBITS(); + break; +#ifdef GUNZIP + case FLAGS: + NEEDBITS(16); + state->flags = (int)(hold); + if ((state->flags & 0xff) != Z_DEFLATED) { + strm->msg = (char *)"unknown compression method"; + state->mode = BAD; + break; + } + if (state->flags & 0xe000) { + strm->msg = (char *)"unknown header flags set"; + state->mode = BAD; + break; + } + if (state->head != Z_NULL) + state->head->text = (int)((hold >> 8) & 1); + if (state->flags & 0x0200) CRC2(state->check, hold); + INITBITS(); + state->mode = TIME; + case TIME: + NEEDBITS(32); + if (state->head != Z_NULL) + state->head->time = hold; + if (state->flags & 0x0200) CRC4(state->check, hold); + INITBITS(); + state->mode = OS; + case OS: + NEEDBITS(16); + if (state->head != Z_NULL) { + state->head->xflags = (int)(hold & 0xff); + state->head->os = (int)(hold >> 8); + } + if (state->flags & 0x0200) CRC2(state->check, hold); + INITBITS(); + state->mode = EXLEN; + case EXLEN: + if (state->flags & 0x0400) { + NEEDBITS(16); + state->length = (unsigned)(hold); + if (state->head != Z_NULL) + state->head->extra_len = (unsigned)hold; + if (state->flags & 0x0200) CRC2(state->check, hold); + INITBITS(); + } + else if (state->head != Z_NULL) + state->head->extra = Z_NULL; + state->mode = EXTRA; + case EXTRA: + if (state->flags & 0x0400) { + copy = state->length; + if (copy > have) copy = have; + if (copy) { + if (state->head != Z_NULL && + state->head->extra != Z_NULL) { + len = state->head->extra_len - state->length; + zmemcpy(state->head->extra + len, next, + len + copy > state->head->extra_max ? + state->head->extra_max - len : copy); + } + if (state->flags & 0x0200) + state->check = crc32(state->check, next, copy); + have -= copy; + next += copy; + state->length -= copy; + } + if (state->length) goto inf_leave; + } + state->length = 0; + state->mode = NAME; + case NAME: + if (state->flags & 0x0800) { + if (have == 0) goto inf_leave; + copy = 0; + do { + len = (unsigned)(next[copy++]); + if (state->head != Z_NULL && + state->head->name != Z_NULL && + state->length < state->head->name_max) + state->head->name[state->length++] = len; + } while (len && copy < have); + if (state->flags & 0x0200) + state->check = crc32(state->check, next, copy); + have -= copy; + next += copy; + if (len) goto inf_leave; + } + else if (state->head != Z_NULL) + state->head->name = Z_NULL; + state->length = 0; + state->mode = COMMENT; + case COMMENT: + if (state->flags & 0x1000) { + if (have == 0) goto inf_leave; + copy = 0; + do { + len = (unsigned)(next[copy++]); + if (state->head != Z_NULL && + state->head->comment != Z_NULL && + state->length < state->head->comm_max) + state->head->comment[state->length++] = len; + } while (len && copy < have); + if (state->flags & 0x0200) + state->check = crc32(state->check, next, copy); + have -= copy; + next += copy; + if (len) goto inf_leave; + } + else if (state->head != Z_NULL) + state->head->comment = Z_NULL; + state->mode = HCRC; + case HCRC: + if (state->flags & 0x0200) { + NEEDBITS(16); + if (hold != (state->check & 0xffff)) { + strm->msg = (char *)"header crc mismatch"; + state->mode = BAD; + break; + } + INITBITS(); + } + if (state->head != Z_NULL) { + state->head->hcrc = (int)((state->flags >> 9) & 1); + state->head->done = 1; + } + strm->adler = state->check = crc32(0L, Z_NULL, 0); + state->mode = TYPE; + break; +#endif + case DICTID: + NEEDBITS(32); + strm->adler = state->check = REVERSE(hold); + INITBITS(); + state->mode = DICT; + case DICT: + if (state->havedict == 0) { + RESTORE(); + return Z_NEED_DICT; + } + strm->adler = state->check = adler32(0L, Z_NULL, 0); + state->mode = TYPE; + case TYPE: + if (flush == Z_BLOCK || flush == Z_TREES) goto inf_leave; + case TYPEDO: + if (state->last) { + BYTEBITS(); + state->mode = CHECK; + break; + } + NEEDBITS(3); + state->last = BITS(1); + DROPBITS(1); + switch (BITS(2)) { + case 0: /* stored block */ + Tracev((stderr, "inflate: stored block%s\n", + state->last ? " (last)" : "")); + state->mode = STORED; + break; + case 1: /* fixed block */ + fixedtables(state); + Tracev((stderr, "inflate: fixed codes block%s\n", + state->last ? " (last)" : "")); + state->mode = LEN_; /* decode codes */ + if (flush == Z_TREES) { + DROPBITS(2); + goto inf_leave; + } + break; + case 2: /* dynamic block */ + Tracev((stderr, "inflate: dynamic codes block%s\n", + state->last ? " (last)" : "")); + state->mode = TABLE; + break; + case 3: + strm->msg = (char *)"invalid block type"; + state->mode = BAD; + } + DROPBITS(2); + break; + case STORED: + BYTEBITS(); /* go to byte boundary */ + NEEDBITS(32); + if ((hold & 0xffff) != ((hold >> 16) ^ 0xffff)) { + strm->msg = (char *)"invalid stored block lengths"; + state->mode = BAD; + break; + } + state->length = (unsigned)hold & 0xffff; + Tracev((stderr, "inflate: stored length %u\n", + state->length)); + INITBITS(); + state->mode = COPY_; + if (flush == Z_TREES) goto inf_leave; + case COPY_: + state->mode = COPY; + case COPY: + copy = state->length; + if (copy) { + if (copy > have) copy = have; + if (copy > left) copy = left; + if (copy == 0) goto inf_leave; + zmemcpy(put, next, copy); + have -= copy; + next += copy; + left -= copy; + put += copy; + state->length -= copy; + break; + } + Tracev((stderr, "inflate: stored end\n")); + state->mode = TYPE; + break; + case TABLE: + NEEDBITS(14); + state->nlen = BITS(5) + 257; + DROPBITS(5); + state->ndist = BITS(5) + 1; + DROPBITS(5); + state->ncode = BITS(4) + 4; + DROPBITS(4); +#ifndef PKZIP_BUG_WORKAROUND + if (state->nlen > 286 || state->ndist > 30) { + strm->msg = (char *)"too many length or distance symbols"; + state->mode = BAD; + break; + } +#endif + Tracev((stderr, "inflate: table sizes ok\n")); + state->have = 0; + state->mode = LENLENS; + case LENLENS: + while (state->have < state->ncode) { + NEEDBITS(3); + state->lens[order[state->have++]] = (unsigned short)BITS(3); + DROPBITS(3); + } + while (state->have < 19) + state->lens[order[state->have++]] = 0; + state->next = state->codes; + state->lencode = (code const FAR *)(state->next); + state->lenbits = 7; + ret = inflate_table(CODES, state->lens, 19, &(state->next), + &(state->lenbits), state->work); + if (ret) { + strm->msg = (char *)"invalid code lengths set"; + state->mode = BAD; + break; + } + Tracev((stderr, "inflate: code lengths ok\n")); + state->have = 0; + state->mode = CODELENS; + case CODELENS: + while (state->have < state->nlen + state->ndist) { + for (;;) { + here = state->lencode[BITS(state->lenbits)]; + if ((unsigned)(here.bits) <= bits) break; + PULLBYTE(); + } + if (here.val < 16) { + NEEDBITS(here.bits); + DROPBITS(here.bits); + state->lens[state->have++] = here.val; + } + else { + if (here.val == 16) { + NEEDBITS(here.bits + 2); + DROPBITS(here.bits); + if (state->have == 0) { + strm->msg = (char *)"invalid bit length repeat"; + state->mode = BAD; + break; + } + len = state->lens[state->have - 1]; + copy = 3 + BITS(2); + DROPBITS(2); + } + else if (here.val == 17) { + NEEDBITS(here.bits + 3); + DROPBITS(here.bits); + len = 0; + copy = 3 + BITS(3); + DROPBITS(3); + } + else { + NEEDBITS(here.bits + 7); + DROPBITS(here.bits); + len = 0; + copy = 11 + BITS(7); + DROPBITS(7); + } + if (state->have + copy > state->nlen + state->ndist) { + strm->msg = (char *)"invalid bit length repeat"; + state->mode = BAD; + break; + } + while (copy--) + state->lens[state->have++] = (unsigned short)len; + } + } + + /* handle error breaks in while */ + if (state->mode == BAD) break; + + /* check for end-of-block code (better have one) */ + if (state->lens[256] == 0) { + strm->msg = (char *)"invalid code -- missing end-of-block"; + state->mode = BAD; + break; + } + + /* build code tables -- note: do not change the lenbits or distbits + values here (9 and 6) without reading the comments in inftrees.h + concerning the ENOUGH constants, which depend on those values */ + state->next = state->codes; + state->lencode = (code const FAR *)(state->next); + state->lenbits = 9; + ret = inflate_table(LENS, state->lens, state->nlen, &(state->next), + &(state->lenbits), state->work); + if (ret) { + strm->msg = (char *)"invalid literal/lengths set"; + state->mode = BAD; + break; + } + state->distcode = (code const FAR *)(state->next); + state->distbits = 6; + ret = inflate_table(DISTS, state->lens + state->nlen, state->ndist, + &(state->next), &(state->distbits), state->work); + if (ret) { + strm->msg = (char *)"invalid distances set"; + state->mode = BAD; + break; + } + Tracev((stderr, "inflate: codes ok\n")); + state->mode = LEN_; + if (flush == Z_TREES) goto inf_leave; + case LEN_: + state->mode = LEN; + case LEN: + if (have >= 6 && left >= 258) { + RESTORE(); + inflate_fast(strm, out); + LOAD(); + if (state->mode == TYPE) + state->back = -1; + break; + } + state->back = 0; + for (;;) { + here = state->lencode[BITS(state->lenbits)]; + if ((unsigned)(here.bits) <= bits) break; + PULLBYTE(); + } + if (here.op && (here.op & 0xf0) == 0) { + last = here; + for (;;) { + here = state->lencode[last.val + + (BITS(last.bits + last.op) >> last.bits)]; + if ((unsigned)(last.bits + here.bits) <= bits) break; + PULLBYTE(); + } + DROPBITS(last.bits); + state->back += last.bits; + } + DROPBITS(here.bits); + state->back += here.bits; + state->length = (unsigned)here.val; + if ((int)(here.op) == 0) { + Tracevv((stderr, here.val >= 0x20 && here.val < 0x7f ? + "inflate: literal '%c'\n" : + "inflate: literal 0x%02x\n", here.val)); + state->mode = LIT; + break; + } + if (here.op & 32) { + Tracevv((stderr, "inflate: end of block\n")); + state->back = -1; + state->mode = TYPE; + break; + } + if (here.op & 64) { + strm->msg = (char *)"invalid literal/length code"; + state->mode = BAD; + break; + } + state->extra = (unsigned)(here.op) & 15; + state->mode = LENEXT; + case LENEXT: + if (state->extra) { + NEEDBITS(state->extra); + state->length += BITS(state->extra); + DROPBITS(state->extra); + state->back += state->extra; + } + Tracevv((stderr, "inflate: length %u\n", state->length)); + state->was = state->length; + state->mode = DIST; + case DIST: + for (;;) { + here = state->distcode[BITS(state->distbits)]; + if ((unsigned)(here.bits) <= bits) break; + PULLBYTE(); + } + if ((here.op & 0xf0) == 0) { + last = here; + for (;;) { + here = state->distcode[last.val + + (BITS(last.bits + last.op) >> last.bits)]; + if ((unsigned)(last.bits + here.bits) <= bits) break; + PULLBYTE(); + } + DROPBITS(last.bits); + state->back += last.bits; + } + DROPBITS(here.bits); + state->back += here.bits; + if (here.op & 64) { + strm->msg = (char *)"invalid distance code"; + state->mode = BAD; + break; + } + state->offset = (unsigned)here.val; + state->extra = (unsigned)(here.op) & 15; + state->mode = DISTEXT; + case DISTEXT: + if (state->extra) { + NEEDBITS(state->extra); + state->offset += BITS(state->extra); + DROPBITS(state->extra); + state->back += state->extra; + } +#ifdef INFLATE_STRICT + if (state->offset > state->dmax) { + strm->msg = (char *)"invalid distance too far back"; + state->mode = BAD; + break; + } +#endif + Tracevv((stderr, "inflate: distance %u\n", state->offset)); + state->mode = MATCH; + case MATCH: + if (left == 0) goto inf_leave; + copy = out - left; + if (state->offset > copy) { /* copy from window */ + copy = state->offset - copy; + if (copy > state->whave) { + if (state->sane) { + strm->msg = (char *)"invalid distance too far back"; + state->mode = BAD; + break; + } +#ifdef INFLATE_ALLOW_INVALID_DISTANCE_TOOFAR_ARRR + Trace((stderr, "inflate.c too far\n")); + copy -= state->whave; + if (copy > state->length) copy = state->length; + if (copy > left) copy = left; + left -= copy; + state->length -= copy; + do { + *put++ = 0; + } while (--copy); + if (state->length == 0) state->mode = LEN; + break; +#endif + } + if (copy > state->wnext) { + copy -= state->wnext; + from = state->window + (state->wsize - copy); + } + else + from = state->window + (state->wnext - copy); + if (copy > state->length) copy = state->length; + } + else { /* copy from output */ + from = put - state->offset; + copy = state->length; + } + if (copy > left) copy = left; + left -= copy; + state->length -= copy; + do { + *put++ = *from++; + } while (--copy); + if (state->length == 0) state->mode = LEN; + break; + case LIT: + if (left == 0) goto inf_leave; + *put++ = (unsigned char)(state->length); + left--; + state->mode = LEN; + break; + case CHECK: + if (state->wrap) { + NEEDBITS(32); + out -= left; + strm->total_out += out; + state->total += out; + if (out) + strm->adler = state->check = + UPDATE(state->check, put - out, out); + out = left; + if (( +#ifdef GUNZIP + state->flags ? hold : +#endif + REVERSE(hold)) != state->check) { + strm->msg = (char *)"incorrect data check"; + state->mode = BAD; + break; + } + INITBITS(); + Tracev((stderr, "inflate: check matches trailer\n")); + } +#ifdef GUNZIP + state->mode = LENGTH; + case LENGTH: + if (state->wrap && state->flags) { + NEEDBITS(32); + if (hold != (state->total & 0xffffffffUL)) { + strm->msg = (char *)"incorrect length check"; + state->mode = BAD; + break; + } + INITBITS(); + Tracev((stderr, "inflate: length matches trailer\n")); + } +#endif + state->mode = DONE; + case DONE: + ret = Z_STREAM_END; + goto inf_leave; + case BAD: + ret = Z_DATA_ERROR; + goto inf_leave; + case MEM: + return Z_MEM_ERROR; + case SYNC: + default: + return Z_STREAM_ERROR; + } + + /* + Return from inflate(), updating the total counts and the check value. + If there was no progress during the inflate() call, return a buffer + error. Call updatewindow() to create and/or update the window state. + Note: a memory error from inflate() is non-recoverable. + */ + inf_leave: + RESTORE(); + if (state->wsize || (state->mode < CHECK && out != strm->avail_out)) + if (updatewindow(strm, out)) { + state->mode = MEM; + return Z_MEM_ERROR; + } + in -= strm->avail_in; + out -= strm->avail_out; + strm->total_in += in; + strm->total_out += out; + state->total += out; + if (state->wrap && out) + strm->adler = state->check = + UPDATE(state->check, strm->next_out - out, out); + strm->data_type = state->bits + (state->last ? 64 : 0) + + (state->mode == TYPE ? 128 : 0) + + (state->mode == LEN_ || state->mode == COPY_ ? 256 : 0); + if (((in == 0 && out == 0) || flush == Z_FINISH) && ret == Z_OK) + ret = Z_BUF_ERROR; + return ret; +} + +int ZEXPORT inflateEnd(strm) +z_streamp strm; +{ + struct inflate_state FAR *state; + if (strm == Z_NULL || strm->state == Z_NULL || strm->zfree == (free_func)0) + return Z_STREAM_ERROR; + state = (struct inflate_state FAR *)strm->state; + if (state->window != Z_NULL) ZFREE(strm, state->window); + ZFREE(strm, strm->state); + strm->state = Z_NULL; + Tracev((stderr, "inflate: end\n")); + return Z_OK; +} + +int ZEXPORT inflateSetDictionary(strm, dictionary, dictLength) +z_streamp strm; +const Bytef *dictionary; +uInt dictLength; +{ + struct inflate_state FAR *state; + unsigned long id; + + /* check state */ + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + state = (struct inflate_state FAR *)strm->state; + if (state->wrap != 0 && state->mode != DICT) + return Z_STREAM_ERROR; + + /* check for correct dictionary id */ + if (state->mode == DICT) { + id = adler32(0L, Z_NULL, 0); + id = adler32(id, dictionary, dictLength); + if (id != state->check) + return Z_DATA_ERROR; + } + + /* copy dictionary to window */ + if (updatewindow(strm, strm->avail_out)) { + state->mode = MEM; + return Z_MEM_ERROR; + } + if (dictLength > state->wsize) { + zmemcpy(state->window, dictionary + dictLength - state->wsize, + state->wsize); + state->whave = state->wsize; + } + else { + zmemcpy(state->window + state->wsize - dictLength, dictionary, + dictLength); + state->whave = dictLength; + } + state->havedict = 1; + Tracev((stderr, "inflate: dictionary set\n")); + return Z_OK; +} + +int ZEXPORT inflateGetHeader(strm, head) +z_streamp strm; +gz_headerp head; +{ + struct inflate_state FAR *state; + + /* check state */ + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + state = (struct inflate_state FAR *)strm->state; + if ((state->wrap & 2) == 0) return Z_STREAM_ERROR; + + /* save header structure */ + state->head = head; + head->done = 0; + return Z_OK; +} + +/* + Search buf[0..len-1] for the pattern: 0, 0, 0xff, 0xff. Return when found + or when out of input. When called, *have is the number of pattern bytes + found in order so far, in 0..3. On return *have is updated to the new + state. If on return *have equals four, then the pattern was found and the + return value is how many bytes were read including the last byte of the + pattern. If *have is less than four, then the pattern has not been found + yet and the return value is len. In the latter case, syncsearch() can be + called again with more data and the *have state. *have is initialized to + zero for the first call. + */ +local unsigned syncsearch(have, buf, len) +unsigned FAR *have; +unsigned char FAR *buf; +unsigned len; +{ + unsigned got; + unsigned next; + + got = *have; + next = 0; + while (next < len && got < 4) { + if ((int)(buf[next]) == (got < 2 ? 0 : 0xff)) + got++; + else if (buf[next]) + got = 0; + else + got = 4 - got; + next++; + } + *have = got; + return next; +} + +int ZEXPORT inflateSync(strm) +z_streamp strm; +{ + unsigned len; /* number of bytes to look at or looked at */ + unsigned long in, out; /* temporary to save total_in and total_out */ + unsigned char buf[4]; /* to restore bit buffer to byte string */ + struct inflate_state FAR *state; + + /* check parameters */ + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + state = (struct inflate_state FAR *)strm->state; + if (strm->avail_in == 0 && state->bits < 8) return Z_BUF_ERROR; + + /* if first time, start search in bit buffer */ + if (state->mode != SYNC) { + state->mode = SYNC; + state->hold <<= state->bits & 7; + state->bits -= state->bits & 7; + len = 0; + while (state->bits >= 8) { + buf[len++] = (unsigned char)(state->hold); + state->hold >>= 8; + state->bits -= 8; + } + state->have = 0; + syncsearch(&(state->have), buf, len); + } + + /* search available input */ + len = syncsearch(&(state->have), strm->next_in, strm->avail_in); + strm->avail_in -= len; + strm->next_in += len; + strm->total_in += len; + + /* return no joy or set up to restart inflate() on a new block */ + if (state->have != 4) return Z_DATA_ERROR; + in = strm->total_in; out = strm->total_out; + inflateReset(strm); + strm->total_in = in; strm->total_out = out; + state->mode = TYPE; + return Z_OK; +} + +/* + Returns true if inflate is currently at the end of a block generated by + Z_SYNC_FLUSH or Z_FULL_FLUSH. This function is used by one PPP + implementation to provide an additional safety check. PPP uses + Z_SYNC_FLUSH but removes the length bytes of the resulting empty stored + block. When decompressing, PPP checks that at the end of input packet, + inflate is waiting for these length bytes. + */ +int ZEXPORT inflateSyncPoint(strm) +z_streamp strm; +{ + struct inflate_state FAR *state; + + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + state = (struct inflate_state FAR *)strm->state; + return state->mode == STORED && state->bits == 0; +} + +int ZEXPORT inflateCopy(dest, source) +z_streamp dest; +z_streamp source; +{ + struct inflate_state FAR *state; + struct inflate_state FAR *copy; + unsigned char FAR *window; + unsigned wsize; + + /* check input */ + if (dest == Z_NULL || source == Z_NULL || source->state == Z_NULL || + source->zalloc == (alloc_func)0 || source->zfree == (free_func)0) + return Z_STREAM_ERROR; + state = (struct inflate_state FAR *)source->state; + + /* allocate space */ + copy = (struct inflate_state FAR *) + ZALLOC(source, 1, sizeof(struct inflate_state)); + if (copy == Z_NULL) return Z_MEM_ERROR; + window = Z_NULL; + if (state->window != Z_NULL) { + window = (unsigned char FAR *) + ZALLOC(source, 1U << state->wbits, sizeof(unsigned char)); + if (window == Z_NULL) { + ZFREE(source, copy); + return Z_MEM_ERROR; + } + } + + /* copy state */ + zmemcpy(dest, source, sizeof(z_stream)); + zmemcpy(copy, state, sizeof(struct inflate_state)); + if (state->lencode >= state->codes && + state->lencode <= state->codes + ENOUGH - 1) { + copy->lencode = copy->codes + (state->lencode - state->codes); + copy->distcode = copy->codes + (state->distcode - state->codes); + } + copy->next = copy->codes + (state->next - state->codes); + if (window != Z_NULL) { + wsize = 1U << state->wbits; + zmemcpy(window, state->window, wsize); + } + copy->window = window; + dest->state = (struct internal_state FAR *)copy; + return Z_OK; +} + +int ZEXPORT inflateUndermine(strm, subvert) +z_streamp strm; +int subvert; +{ + struct inflate_state FAR *state; + + if (strm == Z_NULL || strm->state == Z_NULL) return Z_STREAM_ERROR; + state = (struct inflate_state FAR *)strm->state; + state->sane = !subvert; +#ifdef INFLATE_ALLOW_INVALID_DISTANCE_TOOFAR_ARRR + return Z_OK; +#else + state->sane = 1; + return Z_DATA_ERROR; +#endif +} + +long ZEXPORT inflateMark(strm) +z_streamp strm; +{ + struct inflate_state FAR *state; + + if (strm == Z_NULL || strm->state == Z_NULL) return -1L << 16; + state = (struct inflate_state FAR *)strm->state; + return ((long)(state->back) << 16) + + (state->mode == COPY ? state->length : + (state->mode == MATCH ? state->was - state->length : 0)); +} diff --git a/deps/zlib/inflate.h b/deps/zlib/inflate.h new file mode 100644 index 000000000..95f4986d4 --- /dev/null +++ b/deps/zlib/inflate.h @@ -0,0 +1,122 @@ +/* inflate.h -- internal inflate state definition + * Copyright (C) 1995-2009 Mark Adler + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* WARNING: this file should *not* be used by applications. It is + part of the implementation of the compression library and is + subject to change. Applications should only use zlib.h. + */ + +/* define NO_GZIP when compiling if you want to disable gzip header and + trailer decoding by inflate(). NO_GZIP would be used to avoid linking in + the crc code when it is not needed. For shared libraries, gzip decoding + should be left enabled. */ +#ifndef NO_GZIP +# define GUNZIP +#endif + +/* Possible inflate modes between inflate() calls */ +typedef enum { + HEAD, /* i: waiting for magic header */ + FLAGS, /* i: waiting for method and flags (gzip) */ + TIME, /* i: waiting for modification time (gzip) */ + OS, /* i: waiting for extra flags and operating system (gzip) */ + EXLEN, /* i: waiting for extra length (gzip) */ + EXTRA, /* i: waiting for extra bytes (gzip) */ + NAME, /* i: waiting for end of file name (gzip) */ + COMMENT, /* i: waiting for end of comment (gzip) */ + HCRC, /* i: waiting for header crc (gzip) */ + DICTID, /* i: waiting for dictionary check value */ + DICT, /* waiting for inflateSetDictionary() call */ + TYPE, /* i: waiting for type bits, including last-flag bit */ + TYPEDO, /* i: same, but skip check to exit inflate on new block */ + STORED, /* i: waiting for stored size (length and complement) */ + COPY_, /* i/o: same as COPY below, but only first time in */ + COPY, /* i/o: waiting for input or output to copy stored block */ + TABLE, /* i: waiting for dynamic block table lengths */ + LENLENS, /* i: waiting for code length code lengths */ + CODELENS, /* i: waiting for length/lit and distance code lengths */ + LEN_, /* i: same as LEN below, but only first time in */ + LEN, /* i: waiting for length/lit/eob code */ + LENEXT, /* i: waiting for length extra bits */ + DIST, /* i: waiting for distance code */ + DISTEXT, /* i: waiting for distance extra bits */ + MATCH, /* o: waiting for output space to copy string */ + LIT, /* o: waiting for output space to write literal */ + CHECK, /* i: waiting for 32-bit check value */ + LENGTH, /* i: waiting for 32-bit length (gzip) */ + DONE, /* finished check, done -- remain here until reset */ + BAD, /* got a data error -- remain here until reset */ + MEM, /* got an inflate() memory error -- remain here until reset */ + SYNC /* looking for synchronization bytes to restart inflate() */ +} inflate_mode; + +/* + State transitions between above modes - + + (most modes can go to BAD or MEM on error -- not shown for clarity) + + Process header: + HEAD -> (gzip) or (zlib) or (raw) + (gzip) -> FLAGS -> TIME -> OS -> EXLEN -> EXTRA -> NAME -> COMMENT -> + HCRC -> TYPE + (zlib) -> DICTID or TYPE + DICTID -> DICT -> TYPE + (raw) -> TYPEDO + Read deflate blocks: + TYPE -> TYPEDO -> STORED or TABLE or LEN_ or CHECK + STORED -> COPY_ -> COPY -> TYPE + TABLE -> LENLENS -> CODELENS -> LEN_ + LEN_ -> LEN + Read deflate codes in fixed or dynamic block: + LEN -> LENEXT or LIT or TYPE + LENEXT -> DIST -> DISTEXT -> MATCH -> LEN + LIT -> LEN + Process trailer: + CHECK -> LENGTH -> DONE + */ + +/* state maintained between inflate() calls. Approximately 10K bytes. */ +struct inflate_state { + inflate_mode mode; /* current inflate mode */ + int last; /* true if processing last block */ + int wrap; /* bit 0 true for zlib, bit 1 true for gzip */ + int havedict; /* true if dictionary provided */ + int flags; /* gzip header method and flags (0 if zlib) */ + unsigned dmax; /* zlib header max distance (INFLATE_STRICT) */ + unsigned long check; /* protected copy of check value */ + unsigned long total; /* protected copy of output count */ + gz_headerp head; /* where to save gzip header information */ + /* sliding window */ + unsigned wbits; /* log base 2 of requested window size */ + unsigned wsize; /* window size or zero if not using window */ + unsigned whave; /* valid bytes in the window */ + unsigned wnext; /* window write index */ + unsigned char FAR *window; /* allocated sliding window, if needed */ + /* bit accumulator */ + unsigned long hold; /* input bit accumulator */ + unsigned bits; /* number of bits in "in" */ + /* for string and stored block copying */ + unsigned length; /* literal or length of data to copy */ + unsigned offset; /* distance back to copy string from */ + /* for table and code decoding */ + unsigned extra; /* extra bits needed */ + /* fixed and dynamic code tables */ + code const FAR *lencode; /* starting table for length/literal codes */ + code const FAR *distcode; /* starting table for distance codes */ + unsigned lenbits; /* index bits for lencode */ + unsigned distbits; /* index bits for distcode */ + /* dynamic table building */ + unsigned ncode; /* number of code length code lengths */ + unsigned nlen; /* number of length code lengths */ + unsigned ndist; /* number of distance code lengths */ + unsigned have; /* number of code lengths in lens[] */ + code FAR *next; /* next available space in codes[] */ + unsigned short lens[320]; /* temporary storage for code lengths */ + unsigned short work[288]; /* work area for code table building */ + code codes[ENOUGH]; /* space for code tables */ + int sane; /* if false, allow invalid distance too far */ + int back; /* bits back of last unprocessed length/lit */ + unsigned was; /* initial length of match */ +}; diff --git a/deps/zlib/inftrees.c b/deps/zlib/inftrees.c new file mode 100644 index 000000000..11e9c52ac --- /dev/null +++ b/deps/zlib/inftrees.c @@ -0,0 +1,330 @@ +/* inftrees.c -- generate Huffman trees for efficient decoding + * Copyright (C) 1995-2010 Mark Adler + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +#include "zutil.h" +#include "inftrees.h" + +#define MAXBITS 15 + +const char inflate_copyright[] = + " inflate 1.2.5 Copyright 1995-2010 Mark Adler "; +/* + If you use the zlib library in a product, an acknowledgment is welcome + in the documentation of your product. If for some reason you cannot + include such an acknowledgment, I would appreciate that you keep this + copyright string in the executable of your product. + */ + +/* + Build a set of tables to decode the provided canonical Huffman code. + The code lengths are lens[0..codes-1]. The result starts at *table, + whose indices are 0..2^bits-1. work is a writable array of at least + lens shorts, which is used as a work area. type is the type of code + to be generated, CODES, LENS, or DISTS. On return, zero is success, + -1 is an invalid code, and +1 means that ENOUGH isn't enough. table + on return points to the next available entry's address. bits is the + requested root table index bits, and on return it is the actual root + table index bits. It will differ if the request is greater than the + longest code or if it is less than the shortest code. + */ +int ZLIB_INTERNAL inflate_table(type, lens, codes, table, bits, work) +codetype type; +unsigned short FAR *lens; +unsigned codes; +code FAR * FAR *table; +unsigned FAR *bits; +unsigned short FAR *work; +{ + unsigned len; /* a code's length in bits */ + unsigned sym; /* index of code symbols */ + unsigned min, max; /* minimum and maximum code lengths */ + unsigned root; /* number of index bits for root table */ + unsigned curr; /* number of index bits for current table */ + unsigned drop; /* code bits to drop for sub-table */ + int left; /* number of prefix codes available */ + unsigned used; /* code entries in table used */ + unsigned huff; /* Huffman code */ + unsigned incr; /* for incrementing code, index */ + unsigned fill; /* index for replicating entries */ + unsigned low; /* low bits for current root entry */ + unsigned mask; /* mask for low root bits */ + code here; /* table entry for duplication */ + code FAR *next; /* next available space in table */ + const unsigned short FAR *base; /* base value table to use */ + const unsigned short FAR *extra; /* extra bits table to use */ + int end; /* use base and extra for symbol > end */ + unsigned short count[MAXBITS+1]; /* number of codes of each length */ + unsigned short offs[MAXBITS+1]; /* offsets in table for each length */ + static const unsigned short lbase[31] = { /* Length codes 257..285 base */ + 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 15, 17, 19, 23, 27, 31, + 35, 43, 51, 59, 67, 83, 99, 115, 131, 163, 195, 227, 258, 0, 0}; + static const unsigned short lext[31] = { /* Length codes 257..285 extra */ + 16, 16, 16, 16, 16, 16, 16, 16, 17, 17, 17, 17, 18, 18, 18, 18, + 19, 19, 19, 19, 20, 20, 20, 20, 21, 21, 21, 21, 16, 73, 195}; + static const unsigned short dbase[32] = { /* Distance codes 0..29 base */ + 1, 2, 3, 4, 5, 7, 9, 13, 17, 25, 33, 49, 65, 97, 129, 193, + 257, 385, 513, 769, 1025, 1537, 2049, 3073, 4097, 6145, + 8193, 12289, 16385, 24577, 0, 0}; + static const unsigned short dext[32] = { /* Distance codes 0..29 extra */ + 16, 16, 16, 16, 17, 17, 18, 18, 19, 19, 20, 20, 21, 21, 22, 22, + 23, 23, 24, 24, 25, 25, 26, 26, 27, 27, + 28, 28, 29, 29, 64, 64}; + + /* + Process a set of code lengths to create a canonical Huffman code. The + code lengths are lens[0..codes-1]. Each length corresponds to the + symbols 0..codes-1. The Huffman code is generated by first sorting the + symbols by length from short to long, and retaining the symbol order + for codes with equal lengths. Then the code starts with all zero bits + for the first code of the shortest length, and the codes are integer + increments for the same length, and zeros are appended as the length + increases. For the deflate format, these bits are stored backwards + from their more natural integer increment ordering, and so when the + decoding tables are built in the large loop below, the integer codes + are incremented backwards. + + This routine assumes, but does not check, that all of the entries in + lens[] are in the range 0..MAXBITS. The caller must assure this. + 1..MAXBITS is interpreted as that code length. zero means that that + symbol does not occur in this code. + + The codes are sorted by computing a count of codes for each length, + creating from that a table of starting indices for each length in the + sorted table, and then entering the symbols in order in the sorted + table. The sorted table is work[], with that space being provided by + the caller. + + The length counts are used for other purposes as well, i.e. finding + the minimum and maximum length codes, determining if there are any + codes at all, checking for a valid set of lengths, and looking ahead + at length counts to determine sub-table sizes when building the + decoding tables. + */ + + /* accumulate lengths for codes (assumes lens[] all in 0..MAXBITS) */ + for (len = 0; len <= MAXBITS; len++) + count[len] = 0; + for (sym = 0; sym < codes; sym++) + count[lens[sym]]++; + + /* bound code lengths, force root to be within code lengths */ + root = *bits; + for (max = MAXBITS; max >= 1; max--) + if (count[max] != 0) break; + if (root > max) root = max; + if (max == 0) { /* no symbols to code at all */ + here.op = (unsigned char)64; /* invalid code marker */ + here.bits = (unsigned char)1; + here.val = (unsigned short)0; + *(*table)++ = here; /* make a table to force an error */ + *(*table)++ = here; + *bits = 1; + return 0; /* no symbols, but wait for decoding to report error */ + } + for (min = 1; min < max; min++) + if (count[min] != 0) break; + if (root < min) root = min; + + /* check for an over-subscribed or incomplete set of lengths */ + left = 1; + for (len = 1; len <= MAXBITS; len++) { + left <<= 1; + left -= count[len]; + if (left < 0) return -1; /* over-subscribed */ + } + if (left > 0 && (type == CODES || max != 1)) + return -1; /* incomplete set */ + + /* generate offsets into symbol table for each length for sorting */ + offs[1] = 0; + for (len = 1; len < MAXBITS; len++) + offs[len + 1] = offs[len] + count[len]; + + /* sort symbols by length, by symbol order within each length */ + for (sym = 0; sym < codes; sym++) + if (lens[sym] != 0) work[offs[lens[sym]]++] = (unsigned short)sym; + + /* + Create and fill in decoding tables. In this loop, the table being + filled is at next and has curr index bits. The code being used is huff + with length len. That code is converted to an index by dropping drop + bits off of the bottom. For codes where len is less than drop + curr, + those top drop + curr - len bits are incremented through all values to + fill the table with replicated entries. + + root is the number of index bits for the root table. When len exceeds + root, sub-tables are created pointed to by the root entry with an index + of the low root bits of huff. This is saved in low to check for when a + new sub-table should be started. drop is zero when the root table is + being filled, and drop is root when sub-tables are being filled. + + When a new sub-table is needed, it is necessary to look ahead in the + code lengths to determine what size sub-table is needed. The length + counts are used for this, and so count[] is decremented as codes are + entered in the tables. + + used keeps track of how many table entries have been allocated from the + provided *table space. It is checked for LENS and DIST tables against + the constants ENOUGH_LENS and ENOUGH_DISTS to guard against changes in + the initial root table size constants. See the comments in inftrees.h + for more information. + + sym increments through all symbols, and the loop terminates when + all codes of length max, i.e. all codes, have been processed. This + routine permits incomplete codes, so another loop after this one fills + in the rest of the decoding tables with invalid code markers. + */ + + /* set up for code type */ + switch (type) { + case CODES: + base = extra = work; /* dummy value--not used */ + end = 19; + break; + case LENS: + base = lbase; + base -= 257; + extra = lext; + extra -= 257; + end = 256; + break; + default: /* DISTS */ + base = dbase; + extra = dext; + end = -1; + } + + /* initialize state for loop */ + huff = 0; /* starting code */ + sym = 0; /* starting code symbol */ + len = min; /* starting code length */ + next = *table; /* current table to fill in */ + curr = root; /* current table index bits */ + drop = 0; /* current bits to drop from code for index */ + low = (unsigned)(-1); /* trigger new sub-table when len > root */ + used = 1U << root; /* use root table entries */ + mask = used - 1; /* mask for comparing low */ + + /* check available table space */ + if ((type == LENS && used >= ENOUGH_LENS) || + (type == DISTS && used >= ENOUGH_DISTS)) + return 1; + + /* process all codes and make table entries */ + for (;;) { + /* create table entry */ + here.bits = (unsigned char)(len - drop); + if ((int)(work[sym]) < end) { + here.op = (unsigned char)0; + here.val = work[sym]; + } + else if ((int)(work[sym]) > end) { + here.op = (unsigned char)(extra[work[sym]]); + here.val = base[work[sym]]; + } + else { + here.op = (unsigned char)(32 + 64); /* end of block */ + here.val = 0; + } + + /* replicate for those indices with low len bits equal to huff */ + incr = 1U << (len - drop); + fill = 1U << curr; + min = fill; /* save offset to next table */ + do { + fill -= incr; + next[(huff >> drop) + fill] = here; + } while (fill != 0); + + /* backwards increment the len-bit code huff */ + incr = 1U << (len - 1); + while (huff & incr) + incr >>= 1; + if (incr != 0) { + huff &= incr - 1; + huff += incr; + } + else + huff = 0; + + /* go to next symbol, update count, len */ + sym++; + if (--(count[len]) == 0) { + if (len == max) break; + len = lens[work[sym]]; + } + + /* create new sub-table if needed */ + if (len > root && (huff & mask) != low) { + /* if first time, transition to sub-tables */ + if (drop == 0) + drop = root; + + /* increment past last table */ + next += min; /* here min is 1 << curr */ + + /* determine length of next table */ + curr = len - drop; + left = (int)(1 << curr); + while (curr + drop < max) { + left -= count[curr + drop]; + if (left <= 0) break; + curr++; + left <<= 1; + } + + /* check for enough space */ + used += 1U << curr; + if ((type == LENS && used >= ENOUGH_LENS) || + (type == DISTS && used >= ENOUGH_DISTS)) + return 1; + + /* point entry in root table to sub-table */ + low = huff & mask; + (*table)[low].op = (unsigned char)curr; + (*table)[low].bits = (unsigned char)root; + (*table)[low].val = (unsigned short)(next - *table); + } + } + + /* + Fill in rest of table for incomplete codes. This loop is similar to the + loop above in incrementing huff for table indices. It is assumed that + len is equal to curr + drop, so there is no loop needed to increment + through high index bits. When the current sub-table is filled, the loop + drops back to the root table to fill in any remaining entries there. + */ + here.op = (unsigned char)64; /* invalid code marker */ + here.bits = (unsigned char)(len - drop); + here.val = (unsigned short)0; + while (huff != 0) { + /* when done with sub-table, drop back to root table */ + if (drop != 0 && (huff & mask) != low) { + drop = 0; + len = root; + next = *table; + here.bits = (unsigned char)len; + } + + /* put invalid code marker in table */ + next[huff >> drop] = here; + + /* backwards increment the len-bit code huff */ + incr = 1U << (len - 1); + while (huff & incr) + incr >>= 1; + if (incr != 0) { + huff &= incr - 1; + huff += incr; + } + else + huff = 0; + } + + /* set return parameters */ + *table += used; + *bits = root; + return 0; +} diff --git a/deps/zlib/inftrees.h b/deps/zlib/inftrees.h new file mode 100644 index 000000000..baa53a0b1 --- /dev/null +++ b/deps/zlib/inftrees.h @@ -0,0 +1,62 @@ +/* inftrees.h -- header to use inftrees.c + * Copyright (C) 1995-2005, 2010 Mark Adler + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* WARNING: this file should *not* be used by applications. It is + part of the implementation of the compression library and is + subject to change. Applications should only use zlib.h. + */ + +/* Structure for decoding tables. Each entry provides either the + information needed to do the operation requested by the code that + indexed that table entry, or it provides a pointer to another + table that indexes more bits of the code. op indicates whether + the entry is a pointer to another table, a literal, a length or + distance, an end-of-block, or an invalid code. For a table + pointer, the low four bits of op is the number of index bits of + that table. For a length or distance, the low four bits of op + is the number of extra bits to get after the code. bits is + the number of bits in this code or part of the code to drop off + of the bit buffer. val is the actual byte to output in the case + of a literal, the base length or distance, or the offset from + the current table to the next table. Each entry is four bytes. */ +typedef struct { + unsigned char op; /* operation, extra bits, table bits */ + unsigned char bits; /* bits in this part of the code */ + unsigned short val; /* offset in table or code value */ +} code; + +/* op values as set by inflate_table(): + 00000000 - literal + 0000tttt - table link, tttt != 0 is the number of table index bits + 0001eeee - length or distance, eeee is the number of extra bits + 01100000 - end of block + 01000000 - invalid code + */ + +/* Maximum size of the dynamic table. The maximum number of code structures is + 1444, which is the sum of 852 for literal/length codes and 592 for distance + codes. These values were found by exhaustive searches using the program + examples/enough.c found in the zlib distribtution. The arguments to that + program are the number of symbols, the initial root table size, and the + maximum bit length of a code. "enough 286 9 15" for literal/length codes + returns returns 852, and "enough 30 6 15" for distance codes returns 592. + The initial root table size (9 or 6) is found in the fifth argument of the + inflate_table() calls in inflate.c and infback.c. If the root table size is + changed, then these maximum sizes would be need to be recalculated and + updated. */ +#define ENOUGH_LENS 852 +#define ENOUGH_DISTS 592 +#define ENOUGH (ENOUGH_LENS+ENOUGH_DISTS) + +/* Type of code to build for inflate_table() */ +typedef enum { + CODES, + LENS, + DISTS +} codetype; + +int ZLIB_INTERNAL inflate_table OF((codetype type, unsigned short FAR *lens, + unsigned codes, code FAR * FAR *table, + unsigned FAR *bits, unsigned short FAR *work)); diff --git a/deps/zlib/trees.c b/deps/zlib/trees.c new file mode 100644 index 000000000..56e9bb1c1 --- /dev/null +++ b/deps/zlib/trees.c @@ -0,0 +1,1244 @@ +/* trees.c -- output deflated data using Huffman coding + * Copyright (C) 1995-2010 Jean-loup Gailly + * detect_data_type() function provided freely by Cosmin Truta, 2006 + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* + * ALGORITHM + * + * The "deflation" process uses several Huffman trees. The more + * common source values are represented by shorter bit sequences. + * + * Each code tree is stored in a compressed form which is itself + * a Huffman encoding of the lengths of all the code strings (in + * ascending order by source values). The actual code strings are + * reconstructed from the lengths in the inflate process, as described + * in the deflate specification. + * + * REFERENCES + * + * Deutsch, L.P.,"'Deflate' Compressed Data Format Specification". + * Available in ftp.uu.net:/pub/archiving/zip/doc/deflate-1.1.doc + * + * Storer, James A. + * Data Compression: Methods and Theory, pp. 49-50. + * Computer Science Press, 1988. ISBN 0-7167-8156-5. + * + * Sedgewick, R. + * Algorithms, p290. + * Addison-Wesley, 1983. ISBN 0-201-06672-6. + */ + +/* @(#) $Id$ */ + +/* #define GEN_TREES_H */ + +#include "deflate.h" + +#ifdef DEBUG +# include +#endif + +/* =========================================================================== + * Constants + */ + +#define MAX_BL_BITS 7 +/* Bit length codes must not exceed MAX_BL_BITS bits */ + +#define END_BLOCK 256 +/* end of block literal code */ + +#define REP_3_6 16 +/* repeat previous bit length 3-6 times (2 bits of repeat count) */ + +#define REPZ_3_10 17 +/* repeat a zero length 3-10 times (3 bits of repeat count) */ + +#define REPZ_11_138 18 +/* repeat a zero length 11-138 times (7 bits of repeat count) */ + +local const int extra_lbits[LENGTH_CODES] /* extra bits for each length code */ + = {0,0,0,0,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4,5,5,5,5,0}; + +local const int extra_dbits[D_CODES] /* extra bits for each distance code */ + = {0,0,0,0,1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,10,10,11,11,12,12,13,13}; + +local const int extra_blbits[BL_CODES]/* extra bits for each bit length code */ + = {0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2,3,7}; + +local const uch bl_order[BL_CODES] + = {16,17,18,0,8,7,9,6,10,5,11,4,12,3,13,2,14,1,15}; +/* The lengths of the bit length codes are sent in order of decreasing + * probability, to avoid transmitting the lengths for unused bit length codes. + */ + +#define Buf_size (8 * 2*sizeof(char)) +/* Number of bits used within bi_buf. (bi_buf might be implemented on + * more than 16 bits on some systems.) + */ + +/* =========================================================================== + * Local data. These are initialized only once. + */ + +#define DIST_CODE_LEN 512 /* see definition of array dist_code below */ + +#if defined(GEN_TREES_H) || !defined(STDC) +/* non ANSI compilers may not accept trees.h */ + +local ct_data static_ltree[L_CODES+2]; +/* The static literal tree. Since the bit lengths are imposed, there is no + * need for the L_CODES extra codes used during heap construction. However + * The codes 286 and 287 are needed to build a canonical tree (see _tr_init + * below). + */ + +local ct_data static_dtree[D_CODES]; +/* The static distance tree. (Actually a trivial tree since all codes use + * 5 bits.) + */ + +uch _dist_code[DIST_CODE_LEN]; +/* Distance codes. The first 256 values correspond to the distances + * 3 .. 258, the last 256 values correspond to the top 8 bits of + * the 15 bit distances. + */ + +uch _length_code[MAX_MATCH-MIN_MATCH+1]; +/* length code for each normalized match length (0 == MIN_MATCH) */ + +local int base_length[LENGTH_CODES]; +/* First normalized length for each code (0 = MIN_MATCH) */ + +local int base_dist[D_CODES]; +/* First normalized distance for each code (0 = distance of 1) */ + +#else +# include "trees.h" +#endif /* GEN_TREES_H */ + +struct static_tree_desc_s { + const ct_data *static_tree; /* static tree or NULL */ + const intf *extra_bits; /* extra bits for each code or NULL */ + int extra_base; /* base index for extra_bits */ + int elems; /* max number of elements in the tree */ + int max_length; /* max bit length for the codes */ +}; + +local static_tree_desc static_l_desc = +{static_ltree, extra_lbits, LITERALS+1, L_CODES, MAX_BITS}; + +local static_tree_desc static_d_desc = +{static_dtree, extra_dbits, 0, D_CODES, MAX_BITS}; + +local static_tree_desc static_bl_desc = +{(const ct_data *)0, extra_blbits, 0, BL_CODES, MAX_BL_BITS}; + +/* =========================================================================== + * Local (static) routines in this file. + */ + +local void tr_static_init OF((void)); +local void init_block OF((deflate_state *s)); +local void pqdownheap OF((deflate_state *s, ct_data *tree, int k)); +local void gen_bitlen OF((deflate_state *s, tree_desc *desc)); +local void gen_codes OF((ct_data *tree, int max_code, ushf *bl_count)); +local void build_tree OF((deflate_state *s, tree_desc *desc)); +local void scan_tree OF((deflate_state *s, ct_data *tree, int max_code)); +local void send_tree OF((deflate_state *s, ct_data *tree, int max_code)); +local int build_bl_tree OF((deflate_state *s)); +local void send_all_trees OF((deflate_state *s, int lcodes, int dcodes, + int blcodes)); +local void compress_block OF((deflate_state *s, ct_data *ltree, + ct_data *dtree)); +local int detect_data_type OF((deflate_state *s)); +local unsigned bi_reverse OF((unsigned value, int length)); +local void bi_windup OF((deflate_state *s)); +local void bi_flush OF((deflate_state *s)); +local void copy_block OF((deflate_state *s, charf *buf, unsigned len, + int header)); + +#ifdef GEN_TREES_H +local void gen_trees_header OF((void)); +#endif + +#ifndef DEBUG +# define send_code(s, c, tree) send_bits(s, tree[c].Code, tree[c].Len) + /* Send a code of the given tree. c and tree must not have side effects */ + +#else /* DEBUG */ +# define send_code(s, c, tree) \ + { if (z_verbose>2) fprintf(stderr,"\ncd %3d ",(c)); \ + send_bits(s, tree[c].Code, tree[c].Len); } +#endif + +/* =========================================================================== + * Output a short LSB first on the stream. + * IN assertion: there is enough room in pendingBuf. + */ +#define put_short(s, w) { \ + put_byte(s, (uch)((w) & 0xff)); \ + put_byte(s, (uch)((ush)(w) >> 8)); \ +} + +/* =========================================================================== + * Send a value on a given number of bits. + * IN assertion: length <= 16 and value fits in length bits. + */ +#ifdef DEBUG +local void send_bits OF((deflate_state *s, int value, int length)); + +local void send_bits(s, value, length) + deflate_state *s; + int value; /* value to send */ + int length; /* number of bits */ +{ + Tracevv((stderr," l %2d v %4x ", length, value)); + Assert(length > 0 && length <= 15, "invalid length"); + s->bits_sent += (ulg)length; + + /* If not enough room in bi_buf, use (valid) bits from bi_buf and + * (16 - bi_valid) bits from value, leaving (width - (16-bi_valid)) + * unused bits in value. + */ + if (s->bi_valid > (int)Buf_size - length) { + s->bi_buf |= (ush)value << s->bi_valid; + put_short(s, s->bi_buf); + s->bi_buf = (ush)value >> (Buf_size - s->bi_valid); + s->bi_valid += length - Buf_size; + } else { + s->bi_buf |= (ush)value << s->bi_valid; + s->bi_valid += length; + } +} +#else /* !DEBUG */ + +#define send_bits(s, value, length) \ +{ int len = length;\ + if (s->bi_valid > (int)Buf_size - len) {\ + int val = value;\ + s->bi_buf |= (ush)val << s->bi_valid;\ + put_short(s, s->bi_buf);\ + s->bi_buf = (ush)val >> (Buf_size - s->bi_valid);\ + s->bi_valid += len - Buf_size;\ + } else {\ + s->bi_buf |= (ush)(value) << s->bi_valid;\ + s->bi_valid += len;\ + }\ +} +#endif /* DEBUG */ + + +/* the arguments must not have side effects */ + +/* =========================================================================== + * Initialize the various 'constant' tables. + */ +local void tr_static_init() +{ +#if defined(GEN_TREES_H) || !defined(STDC) + static int static_init_done = 0; + int n; /* iterates over tree elements */ + int bits; /* bit counter */ + int length; /* length value */ + int code; /* code value */ + int dist; /* distance index */ + ush bl_count[MAX_BITS+1]; + /* number of codes at each bit length for an optimal tree */ + + if (static_init_done) return; + + /* For some embedded targets, global variables are not initialized: */ +#ifdef NO_INIT_GLOBAL_POINTERS + static_l_desc.static_tree = static_ltree; + static_l_desc.extra_bits = extra_lbits; + static_d_desc.static_tree = static_dtree; + static_d_desc.extra_bits = extra_dbits; + static_bl_desc.extra_bits = extra_blbits; +#endif + + /* Initialize the mapping length (0..255) -> length code (0..28) */ + length = 0; + for (code = 0; code < LENGTH_CODES-1; code++) { + base_length[code] = length; + for (n = 0; n < (1< dist code (0..29) */ + dist = 0; + for (code = 0 ; code < 16; code++) { + base_dist[code] = dist; + for (n = 0; n < (1<>= 7; /* from now on, all distances are divided by 128 */ + for ( ; code < D_CODES; code++) { + base_dist[code] = dist << 7; + for (n = 0; n < (1<<(extra_dbits[code]-7)); n++) { + _dist_code[256 + dist++] = (uch)code; + } + } + Assert (dist == 256, "tr_static_init: 256+dist != 512"); + + /* Construct the codes of the static literal tree */ + for (bits = 0; bits <= MAX_BITS; bits++) bl_count[bits] = 0; + n = 0; + while (n <= 143) static_ltree[n++].Len = 8, bl_count[8]++; + while (n <= 255) static_ltree[n++].Len = 9, bl_count[9]++; + while (n <= 279) static_ltree[n++].Len = 7, bl_count[7]++; + while (n <= 287) static_ltree[n++].Len = 8, bl_count[8]++; + /* Codes 286 and 287 do not exist, but we must include them in the + * tree construction to get a canonical Huffman tree (longest code + * all ones) + */ + gen_codes((ct_data *)static_ltree, L_CODES+1, bl_count); + + /* The static distance tree is trivial: */ + for (n = 0; n < D_CODES; n++) { + static_dtree[n].Len = 5; + static_dtree[n].Code = bi_reverse((unsigned)n, 5); + } + static_init_done = 1; + +# ifdef GEN_TREES_H + gen_trees_header(); +# endif +#endif /* defined(GEN_TREES_H) || !defined(STDC) */ +} + +/* =========================================================================== + * Genererate the file trees.h describing the static trees. + */ +#ifdef GEN_TREES_H +# ifndef DEBUG +# include +# endif + +# define SEPARATOR(i, last, width) \ + ((i) == (last)? "\n};\n\n" : \ + ((i) % (width) == (width)-1 ? ",\n" : ", ")) + +void gen_trees_header() +{ + FILE *header = fopen("trees.h", "w"); + int i; + + Assert (header != NULL, "Can't open trees.h"); + fprintf(header, + "/* header created automatically with -DGEN_TREES_H */\n\n"); + + fprintf(header, "local const ct_data static_ltree[L_CODES+2] = {\n"); + for (i = 0; i < L_CODES+2; i++) { + fprintf(header, "{{%3u},{%3u}}%s", static_ltree[i].Code, + static_ltree[i].Len, SEPARATOR(i, L_CODES+1, 5)); + } + + fprintf(header, "local const ct_data static_dtree[D_CODES] = {\n"); + for (i = 0; i < D_CODES; i++) { + fprintf(header, "{{%2u},{%2u}}%s", static_dtree[i].Code, + static_dtree[i].Len, SEPARATOR(i, D_CODES-1, 5)); + } + + fprintf(header, "const uch ZLIB_INTERNAL _dist_code[DIST_CODE_LEN] = {\n"); + for (i = 0; i < DIST_CODE_LEN; i++) { + fprintf(header, "%2u%s", _dist_code[i], + SEPARATOR(i, DIST_CODE_LEN-1, 20)); + } + + fprintf(header, + "const uch ZLIB_INTERNAL _length_code[MAX_MATCH-MIN_MATCH+1]= {\n"); + for (i = 0; i < MAX_MATCH-MIN_MATCH+1; i++) { + fprintf(header, "%2u%s", _length_code[i], + SEPARATOR(i, MAX_MATCH-MIN_MATCH, 20)); + } + + fprintf(header, "local const int base_length[LENGTH_CODES] = {\n"); + for (i = 0; i < LENGTH_CODES; i++) { + fprintf(header, "%1u%s", base_length[i], + SEPARATOR(i, LENGTH_CODES-1, 20)); + } + + fprintf(header, "local const int base_dist[D_CODES] = {\n"); + for (i = 0; i < D_CODES; i++) { + fprintf(header, "%5u%s", base_dist[i], + SEPARATOR(i, D_CODES-1, 10)); + } + + fclose(header); +} +#endif /* GEN_TREES_H */ + +/* =========================================================================== + * Initialize the tree data structures for a new zlib stream. + */ +void ZLIB_INTERNAL _tr_init(s) + deflate_state *s; +{ + tr_static_init(); + + s->l_desc.dyn_tree = s->dyn_ltree; + s->l_desc.stat_desc = &static_l_desc; + + s->d_desc.dyn_tree = s->dyn_dtree; + s->d_desc.stat_desc = &static_d_desc; + + s->bl_desc.dyn_tree = s->bl_tree; + s->bl_desc.stat_desc = &static_bl_desc; + + s->bi_buf = 0; + s->bi_valid = 0; + s->last_eob_len = 8; /* enough lookahead for inflate */ +#ifdef DEBUG + s->compressed_len = 0L; + s->bits_sent = 0L; +#endif + + /* Initialize the first block of the first file: */ + init_block(s); +} + +/* =========================================================================== + * Initialize a new block. + */ +local void init_block(s) + deflate_state *s; +{ + int n; /* iterates over tree elements */ + + /* Initialize the trees. */ + for (n = 0; n < L_CODES; n++) s->dyn_ltree[n].Freq = 0; + for (n = 0; n < D_CODES; n++) s->dyn_dtree[n].Freq = 0; + for (n = 0; n < BL_CODES; n++) s->bl_tree[n].Freq = 0; + + s->dyn_ltree[END_BLOCK].Freq = 1; + s->opt_len = s->static_len = 0L; + s->last_lit = s->matches = 0; +} + +#define SMALLEST 1 +/* Index within the heap array of least frequent node in the Huffman tree */ + + +/* =========================================================================== + * Remove the smallest element from the heap and recreate the heap with + * one less element. Updates heap and heap_len. + */ +#define pqremove(s, tree, top) \ +{\ + top = s->heap[SMALLEST]; \ + s->heap[SMALLEST] = s->heap[s->heap_len--]; \ + pqdownheap(s, tree, SMALLEST); \ +} + +/* =========================================================================== + * Compares to subtrees, using the tree depth as tie breaker when + * the subtrees have equal frequency. This minimizes the worst case length. + */ +#define smaller(tree, n, m, depth) \ + (tree[n].Freq < tree[m].Freq || \ + (tree[n].Freq == tree[m].Freq && depth[n] <= depth[m])) + +/* =========================================================================== + * Restore the heap property by moving down the tree starting at node k, + * exchanging a node with the smallest of its two sons if necessary, stopping + * when the heap property is re-established (each father smaller than its + * two sons). + */ +local void pqdownheap(s, tree, k) + deflate_state *s; + ct_data *tree; /* the tree to restore */ + int k; /* node to move down */ +{ + int v = s->heap[k]; + int j = k << 1; /* left son of k */ + while (j <= s->heap_len) { + /* Set j to the smallest of the two sons: */ + if (j < s->heap_len && + smaller(tree, s->heap[j+1], s->heap[j], s->depth)) { + j++; + } + /* Exit if v is smaller than both sons */ + if (smaller(tree, v, s->heap[j], s->depth)) break; + + /* Exchange v with the smallest son */ + s->heap[k] = s->heap[j]; k = j; + + /* And continue down the tree, setting j to the left son of k */ + j <<= 1; + } + s->heap[k] = v; +} + +/* =========================================================================== + * Compute the optimal bit lengths for a tree and update the total bit length + * for the current block. + * IN assertion: the fields freq and dad are set, heap[heap_max] and + * above are the tree nodes sorted by increasing frequency. + * OUT assertions: the field len is set to the optimal bit length, the + * array bl_count contains the frequencies for each bit length. + * The length opt_len is updated; static_len is also updated if stree is + * not null. + */ +local void gen_bitlen(s, desc) + deflate_state *s; + tree_desc *desc; /* the tree descriptor */ +{ + ct_data *tree = desc->dyn_tree; + int max_code = desc->max_code; + const ct_data *stree = desc->stat_desc->static_tree; + const intf *extra = desc->stat_desc->extra_bits; + int base = desc->stat_desc->extra_base; + int max_length = desc->stat_desc->max_length; + int h; /* heap index */ + int n, m; /* iterate over the tree elements */ + int bits; /* bit length */ + int xbits; /* extra bits */ + ush f; /* frequency */ + int overflow = 0; /* number of elements with bit length too large */ + + for (bits = 0; bits <= MAX_BITS; bits++) s->bl_count[bits] = 0; + + /* In a first pass, compute the optimal bit lengths (which may + * overflow in the case of the bit length tree). + */ + tree[s->heap[s->heap_max]].Len = 0; /* root of the heap */ + + for (h = s->heap_max+1; h < HEAP_SIZE; h++) { + n = s->heap[h]; + bits = tree[tree[n].Dad].Len + 1; + if (bits > max_length) bits = max_length, overflow++; + tree[n].Len = (ush)bits; + /* We overwrite tree[n].Dad which is no longer needed */ + + if (n > max_code) continue; /* not a leaf node */ + + s->bl_count[bits]++; + xbits = 0; + if (n >= base) xbits = extra[n-base]; + f = tree[n].Freq; + s->opt_len += (ulg)f * (bits + xbits); + if (stree) s->static_len += (ulg)f * (stree[n].Len + xbits); + } + if (overflow == 0) return; + + Trace((stderr,"\nbit length overflow\n")); + /* This happens for example on obj2 and pic of the Calgary corpus */ + + /* Find the first bit length which could increase: */ + do { + bits = max_length-1; + while (s->bl_count[bits] == 0) bits--; + s->bl_count[bits]--; /* move one leaf down the tree */ + s->bl_count[bits+1] += 2; /* move one overflow item as its brother */ + s->bl_count[max_length]--; + /* The brother of the overflow item also moves one step up, + * but this does not affect bl_count[max_length] + */ + overflow -= 2; + } while (overflow > 0); + + /* Now recompute all bit lengths, scanning in increasing frequency. + * h is still equal to HEAP_SIZE. (It is simpler to reconstruct all + * lengths instead of fixing only the wrong ones. This idea is taken + * from 'ar' written by Haruhiko Okumura.) + */ + for (bits = max_length; bits != 0; bits--) { + n = s->bl_count[bits]; + while (n != 0) { + m = s->heap[--h]; + if (m > max_code) continue; + if ((unsigned) tree[m].Len != (unsigned) bits) { + Trace((stderr,"code %d bits %d->%d\n", m, tree[m].Len, bits)); + s->opt_len += ((long)bits - (long)tree[m].Len) + *(long)tree[m].Freq; + tree[m].Len = (ush)bits; + } + n--; + } + } +} + +/* =========================================================================== + * Generate the codes for a given tree and bit counts (which need not be + * optimal). + * IN assertion: the array bl_count contains the bit length statistics for + * the given tree and the field len is set for all tree elements. + * OUT assertion: the field code is set for all tree elements of non + * zero code length. + */ +local void gen_codes (tree, max_code, bl_count) + ct_data *tree; /* the tree to decorate */ + int max_code; /* largest code with non zero frequency */ + ushf *bl_count; /* number of codes at each bit length */ +{ + ush next_code[MAX_BITS+1]; /* next code value for each bit length */ + ush code = 0; /* running code value */ + int bits; /* bit index */ + int n; /* code index */ + + /* The distribution counts are first used to generate the code values + * without bit reversal. + */ + for (bits = 1; bits <= MAX_BITS; bits++) { + next_code[bits] = code = (code + bl_count[bits-1]) << 1; + } + /* Check that the bit counts in bl_count are consistent. The last code + * must be all ones. + */ + Assert (code + bl_count[MAX_BITS]-1 == (1<dyn_tree; + const ct_data *stree = desc->stat_desc->static_tree; + int elems = desc->stat_desc->elems; + int n, m; /* iterate over heap elements */ + int max_code = -1; /* largest code with non zero frequency */ + int node; /* new node being created */ + + /* Construct the initial heap, with least frequent element in + * heap[SMALLEST]. The sons of heap[n] are heap[2*n] and heap[2*n+1]. + * heap[0] is not used. + */ + s->heap_len = 0, s->heap_max = HEAP_SIZE; + + for (n = 0; n < elems; n++) { + if (tree[n].Freq != 0) { + s->heap[++(s->heap_len)] = max_code = n; + s->depth[n] = 0; + } else { + tree[n].Len = 0; + } + } + + /* The pkzip format requires that at least one distance code exists, + * and that at least one bit should be sent even if there is only one + * possible code. So to avoid special checks later on we force at least + * two codes of non zero frequency. + */ + while (s->heap_len < 2) { + node = s->heap[++(s->heap_len)] = (max_code < 2 ? ++max_code : 0); + tree[node].Freq = 1; + s->depth[node] = 0; + s->opt_len--; if (stree) s->static_len -= stree[node].Len; + /* node is 0 or 1 so it does not have extra bits */ + } + desc->max_code = max_code; + + /* The elements heap[heap_len/2+1 .. heap_len] are leaves of the tree, + * establish sub-heaps of increasing lengths: + */ + for (n = s->heap_len/2; n >= 1; n--) pqdownheap(s, tree, n); + + /* Construct the Huffman tree by repeatedly combining the least two + * frequent nodes. + */ + node = elems; /* next internal node of the tree */ + do { + pqremove(s, tree, n); /* n = node of least frequency */ + m = s->heap[SMALLEST]; /* m = node of next least frequency */ + + s->heap[--(s->heap_max)] = n; /* keep the nodes sorted by frequency */ + s->heap[--(s->heap_max)] = m; + + /* Create a new node father of n and m */ + tree[node].Freq = tree[n].Freq + tree[m].Freq; + s->depth[node] = (uch)((s->depth[n] >= s->depth[m] ? + s->depth[n] : s->depth[m]) + 1); + tree[n].Dad = tree[m].Dad = (ush)node; +#ifdef DUMP_BL_TREE + if (tree == s->bl_tree) { + fprintf(stderr,"\nnode %d(%d), sons %d(%d) %d(%d)", + node, tree[node].Freq, n, tree[n].Freq, m, tree[m].Freq); + } +#endif + /* and insert the new node in the heap */ + s->heap[SMALLEST] = node++; + pqdownheap(s, tree, SMALLEST); + + } while (s->heap_len >= 2); + + s->heap[--(s->heap_max)] = s->heap[SMALLEST]; + + /* At this point, the fields freq and dad are set. We can now + * generate the bit lengths. + */ + gen_bitlen(s, (tree_desc *)desc); + + /* The field len is now set, we can generate the bit codes */ + gen_codes ((ct_data *)tree, max_code, s->bl_count); +} + +/* =========================================================================== + * Scan a literal or distance tree to determine the frequencies of the codes + * in the bit length tree. + */ +local void scan_tree (s, tree, max_code) + deflate_state *s; + ct_data *tree; /* the tree to be scanned */ + int max_code; /* and its largest code of non zero frequency */ +{ + int n; /* iterates over all tree elements */ + int prevlen = -1; /* last emitted length */ + int curlen; /* length of current code */ + int nextlen = tree[0].Len; /* length of next code */ + int count = 0; /* repeat count of the current code */ + int max_count = 7; /* max repeat count */ + int min_count = 4; /* min repeat count */ + + if (nextlen == 0) max_count = 138, min_count = 3; + tree[max_code+1].Len = (ush)0xffff; /* guard */ + + for (n = 0; n <= max_code; n++) { + curlen = nextlen; nextlen = tree[n+1].Len; + if (++count < max_count && curlen == nextlen) { + continue; + } else if (count < min_count) { + s->bl_tree[curlen].Freq += count; + } else if (curlen != 0) { + if (curlen != prevlen) s->bl_tree[curlen].Freq++; + s->bl_tree[REP_3_6].Freq++; + } else if (count <= 10) { + s->bl_tree[REPZ_3_10].Freq++; + } else { + s->bl_tree[REPZ_11_138].Freq++; + } + count = 0; prevlen = curlen; + if (nextlen == 0) { + max_count = 138, min_count = 3; + } else if (curlen == nextlen) { + max_count = 6, min_count = 3; + } else { + max_count = 7, min_count = 4; + } + } +} + +/* =========================================================================== + * Send a literal or distance tree in compressed form, using the codes in + * bl_tree. + */ +local void send_tree (s, tree, max_code) + deflate_state *s; + ct_data *tree; /* the tree to be scanned */ + int max_code; /* and its largest code of non zero frequency */ +{ + int n; /* iterates over all tree elements */ + int prevlen = -1; /* last emitted length */ + int curlen; /* length of current code */ + int nextlen = tree[0].Len; /* length of next code */ + int count = 0; /* repeat count of the current code */ + int max_count = 7; /* max repeat count */ + int min_count = 4; /* min repeat count */ + + /* tree[max_code+1].Len = -1; */ /* guard already set */ + if (nextlen == 0) max_count = 138, min_count = 3; + + for (n = 0; n <= max_code; n++) { + curlen = nextlen; nextlen = tree[n+1].Len; + if (++count < max_count && curlen == nextlen) { + continue; + } else if (count < min_count) { + do { send_code(s, curlen, s->bl_tree); } while (--count != 0); + + } else if (curlen != 0) { + if (curlen != prevlen) { + send_code(s, curlen, s->bl_tree); count--; + } + Assert(count >= 3 && count <= 6, " 3_6?"); + send_code(s, REP_3_6, s->bl_tree); send_bits(s, count-3, 2); + + } else if (count <= 10) { + send_code(s, REPZ_3_10, s->bl_tree); send_bits(s, count-3, 3); + + } else { + send_code(s, REPZ_11_138, s->bl_tree); send_bits(s, count-11, 7); + } + count = 0; prevlen = curlen; + if (nextlen == 0) { + max_count = 138, min_count = 3; + } else if (curlen == nextlen) { + max_count = 6, min_count = 3; + } else { + max_count = 7, min_count = 4; + } + } +} + +/* =========================================================================== + * Construct the Huffman tree for the bit lengths and return the index in + * bl_order of the last bit length code to send. + */ +local int build_bl_tree(s) + deflate_state *s; +{ + int max_blindex; /* index of last bit length code of non zero freq */ + + /* Determine the bit length frequencies for literal and distance trees */ + scan_tree(s, (ct_data *)s->dyn_ltree, s->l_desc.max_code); + scan_tree(s, (ct_data *)s->dyn_dtree, s->d_desc.max_code); + + /* Build the bit length tree: */ + build_tree(s, (tree_desc *)(&(s->bl_desc))); + /* opt_len now includes the length of the tree representations, except + * the lengths of the bit lengths codes and the 5+5+4 bits for the counts. + */ + + /* Determine the number of bit length codes to send. The pkzip format + * requires that at least 4 bit length codes be sent. (appnote.txt says + * 3 but the actual value used is 4.) + */ + for (max_blindex = BL_CODES-1; max_blindex >= 3; max_blindex--) { + if (s->bl_tree[bl_order[max_blindex]].Len != 0) break; + } + /* Update opt_len to include the bit length tree and counts */ + s->opt_len += 3*(max_blindex+1) + 5+5+4; + Tracev((stderr, "\ndyn trees: dyn %ld, stat %ld", + s->opt_len, s->static_len)); + + return max_blindex; +} + +/* =========================================================================== + * Send the header for a block using dynamic Huffman trees: the counts, the + * lengths of the bit length codes, the literal tree and the distance tree. + * IN assertion: lcodes >= 257, dcodes >= 1, blcodes >= 4. + */ +local void send_all_trees(s, lcodes, dcodes, blcodes) + deflate_state *s; + int lcodes, dcodes, blcodes; /* number of codes for each tree */ +{ + int rank; /* index in bl_order */ + + Assert (lcodes >= 257 && dcodes >= 1 && blcodes >= 4, "not enough codes"); + Assert (lcodes <= L_CODES && dcodes <= D_CODES && blcodes <= BL_CODES, + "too many codes"); + Tracev((stderr, "\nbl counts: ")); + send_bits(s, lcodes-257, 5); /* not +255 as stated in appnote.txt */ + send_bits(s, dcodes-1, 5); + send_bits(s, blcodes-4, 4); /* not -3 as stated in appnote.txt */ + for (rank = 0; rank < blcodes; rank++) { + Tracev((stderr, "\nbl code %2d ", bl_order[rank])); + send_bits(s, s->bl_tree[bl_order[rank]].Len, 3); + } + Tracev((stderr, "\nbl tree: sent %ld", s->bits_sent)); + + send_tree(s, (ct_data *)s->dyn_ltree, lcodes-1); /* literal tree */ + Tracev((stderr, "\nlit tree: sent %ld", s->bits_sent)); + + send_tree(s, (ct_data *)s->dyn_dtree, dcodes-1); /* distance tree */ + Tracev((stderr, "\ndist tree: sent %ld", s->bits_sent)); +} + +/* =========================================================================== + * Send a stored block + */ +void ZLIB_INTERNAL _tr_stored_block(s, buf, stored_len, last) + deflate_state *s; + charf *buf; /* input block */ + ulg stored_len; /* length of input block */ + int last; /* one if this is the last block for a file */ +{ + send_bits(s, (STORED_BLOCK<<1)+last, 3); /* send block type */ +#ifdef DEBUG + s->compressed_len = (s->compressed_len + 3 + 7) & (ulg)~7L; + s->compressed_len += (stored_len + 4) << 3; +#endif + copy_block(s, buf, (unsigned)stored_len, 1); /* with header */ +} + +/* =========================================================================== + * Send one empty static block to give enough lookahead for inflate. + * This takes 10 bits, of which 7 may remain in the bit buffer. + * The current inflate code requires 9 bits of lookahead. If the + * last two codes for the previous block (real code plus EOB) were coded + * on 5 bits or less, inflate may have only 5+3 bits of lookahead to decode + * the last real code. In this case we send two empty static blocks instead + * of one. (There are no problems if the previous block is stored or fixed.) + * To simplify the code, we assume the worst case of last real code encoded + * on one bit only. + */ +void ZLIB_INTERNAL _tr_align(s) + deflate_state *s; +{ + send_bits(s, STATIC_TREES<<1, 3); + send_code(s, END_BLOCK, static_ltree); +#ifdef DEBUG + s->compressed_len += 10L; /* 3 for block type, 7 for EOB */ +#endif + bi_flush(s); + /* Of the 10 bits for the empty block, we have already sent + * (10 - bi_valid) bits. The lookahead for the last real code (before + * the EOB of the previous block) was thus at least one plus the length + * of the EOB plus what we have just sent of the empty static block. + */ + if (1 + s->last_eob_len + 10 - s->bi_valid < 9) { + send_bits(s, STATIC_TREES<<1, 3); + send_code(s, END_BLOCK, static_ltree); +#ifdef DEBUG + s->compressed_len += 10L; +#endif + bi_flush(s); + } + s->last_eob_len = 7; +} + +/* =========================================================================== + * Determine the best encoding for the current block: dynamic trees, static + * trees or store, and output the encoded block to the zip file. + */ +void ZLIB_INTERNAL _tr_flush_block(s, buf, stored_len, last) + deflate_state *s; + charf *buf; /* input block, or NULL if too old */ + ulg stored_len; /* length of input block */ + int last; /* one if this is the last block for a file */ +{ + ulg opt_lenb, static_lenb; /* opt_len and static_len in bytes */ + int max_blindex = 0; /* index of last bit length code of non zero freq */ + + /* Build the Huffman trees unless a stored block is forced */ + if (s->level > 0) { + + /* Check if the file is binary or text */ + if (s->strm->data_type == Z_UNKNOWN) + s->strm->data_type = detect_data_type(s); + + /* Construct the literal and distance trees */ + build_tree(s, (tree_desc *)(&(s->l_desc))); + Tracev((stderr, "\nlit data: dyn %ld, stat %ld", s->opt_len, + s->static_len)); + + build_tree(s, (tree_desc *)(&(s->d_desc))); + Tracev((stderr, "\ndist data: dyn %ld, stat %ld", s->opt_len, + s->static_len)); + /* At this point, opt_len and static_len are the total bit lengths of + * the compressed block data, excluding the tree representations. + */ + + /* Build the bit length tree for the above two trees, and get the index + * in bl_order of the last bit length code to send. + */ + max_blindex = build_bl_tree(s); + + /* Determine the best encoding. Compute the block lengths in bytes. */ + opt_lenb = (s->opt_len+3+7)>>3; + static_lenb = (s->static_len+3+7)>>3; + + Tracev((stderr, "\nopt %lu(%lu) stat %lu(%lu) stored %lu lit %u ", + opt_lenb, s->opt_len, static_lenb, s->static_len, stored_len, + s->last_lit)); + + if (static_lenb <= opt_lenb) opt_lenb = static_lenb; + + } else { + Assert(buf != (char*)0, "lost buf"); + opt_lenb = static_lenb = stored_len + 5; /* force a stored block */ + } + +#ifdef FORCE_STORED + if (buf != (char*)0) { /* force stored block */ +#else + if (stored_len+4 <= opt_lenb && buf != (char*)0) { + /* 4: two words for the lengths */ +#endif + /* The test buf != NULL is only necessary if LIT_BUFSIZE > WSIZE. + * Otherwise we can't have processed more than WSIZE input bytes since + * the last block flush, because compression would have been + * successful. If LIT_BUFSIZE <= WSIZE, it is never too late to + * transform a block into a stored block. + */ + _tr_stored_block(s, buf, stored_len, last); + +#ifdef FORCE_STATIC + } else if (static_lenb >= 0) { /* force static trees */ +#else + } else if (s->strategy == Z_FIXED || static_lenb == opt_lenb) { +#endif + send_bits(s, (STATIC_TREES<<1)+last, 3); + compress_block(s, (ct_data *)static_ltree, (ct_data *)static_dtree); +#ifdef DEBUG + s->compressed_len += 3 + s->static_len; +#endif + } else { + send_bits(s, (DYN_TREES<<1)+last, 3); + send_all_trees(s, s->l_desc.max_code+1, s->d_desc.max_code+1, + max_blindex+1); + compress_block(s, (ct_data *)s->dyn_ltree, (ct_data *)s->dyn_dtree); +#ifdef DEBUG + s->compressed_len += 3 + s->opt_len; +#endif + } + Assert (s->compressed_len == s->bits_sent, "bad compressed size"); + /* The above check is made mod 2^32, for files larger than 512 MB + * and uLong implemented on 32 bits. + */ + init_block(s); + + if (last) { + bi_windup(s); +#ifdef DEBUG + s->compressed_len += 7; /* align on byte boundary */ +#endif + } + Tracev((stderr,"\ncomprlen %lu(%lu) ", s->compressed_len>>3, + s->compressed_len-7*last)); +} + +/* =========================================================================== + * Save the match info and tally the frequency counts. Return true if + * the current block must be flushed. + */ +int ZLIB_INTERNAL _tr_tally (s, dist, lc) + deflate_state *s; + unsigned dist; /* distance of matched string */ + unsigned lc; /* match length-MIN_MATCH or unmatched char (if dist==0) */ +{ + s->d_buf[s->last_lit] = (ush)dist; + s->l_buf[s->last_lit++] = (uch)lc; + if (dist == 0) { + /* lc is the unmatched char */ + s->dyn_ltree[lc].Freq++; + } else { + s->matches++; + /* Here, lc is the match length - MIN_MATCH */ + dist--; /* dist = match distance - 1 */ + Assert((ush)dist < (ush)MAX_DIST(s) && + (ush)lc <= (ush)(MAX_MATCH-MIN_MATCH) && + (ush)d_code(dist) < (ush)D_CODES, "_tr_tally: bad match"); + + s->dyn_ltree[_length_code[lc]+LITERALS+1].Freq++; + s->dyn_dtree[d_code(dist)].Freq++; + } + +#ifdef TRUNCATE_BLOCK + /* Try to guess if it is profitable to stop the current block here */ + if ((s->last_lit & 0x1fff) == 0 && s->level > 2) { + /* Compute an upper bound for the compressed length */ + ulg out_length = (ulg)s->last_lit*8L; + ulg in_length = (ulg)((long)s->strstart - s->block_start); + int dcode; + for (dcode = 0; dcode < D_CODES; dcode++) { + out_length += (ulg)s->dyn_dtree[dcode].Freq * + (5L+extra_dbits[dcode]); + } + out_length >>= 3; + Tracev((stderr,"\nlast_lit %u, in %ld, out ~%ld(%ld%%) ", + s->last_lit, in_length, out_length, + 100L - out_length*100L/in_length)); + if (s->matches < s->last_lit/2 && out_length < in_length/2) return 1; + } +#endif + return (s->last_lit == s->lit_bufsize-1); + /* We avoid equality with lit_bufsize because of wraparound at 64K + * on 16 bit machines and because stored blocks are restricted to + * 64K-1 bytes. + */ +} + +/* =========================================================================== + * Send the block data compressed using the given Huffman trees + */ +local void compress_block(s, ltree, dtree) + deflate_state *s; + ct_data *ltree; /* literal tree */ + ct_data *dtree; /* distance tree */ +{ + unsigned dist; /* distance of matched string */ + int lc; /* match length or unmatched char (if dist == 0) */ + unsigned lx = 0; /* running index in l_buf */ + unsigned code; /* the code to send */ + int extra; /* number of extra bits to send */ + + if (s->last_lit != 0) do { + dist = s->d_buf[lx]; + lc = s->l_buf[lx++]; + if (dist == 0) { + send_code(s, lc, ltree); /* send a literal byte */ + Tracecv(isgraph(lc), (stderr," '%c' ", lc)); + } else { + /* Here, lc is the match length - MIN_MATCH */ + code = _length_code[lc]; + send_code(s, code+LITERALS+1, ltree); /* send the length code */ + extra = extra_lbits[code]; + if (extra != 0) { + lc -= base_length[code]; + send_bits(s, lc, extra); /* send the extra length bits */ + } + dist--; /* dist is now the match distance - 1 */ + code = d_code(dist); + Assert (code < D_CODES, "bad d_code"); + + send_code(s, code, dtree); /* send the distance code */ + extra = extra_dbits[code]; + if (extra != 0) { + dist -= base_dist[code]; + send_bits(s, dist, extra); /* send the extra distance bits */ + } + } /* literal or match pair ? */ + + /* Check that the overlay between pending_buf and d_buf+l_buf is ok: */ + Assert((uInt)(s->pending) < s->lit_bufsize + 2*lx, + "pendingBuf overflow"); + + } while (lx < s->last_lit); + + send_code(s, END_BLOCK, ltree); + s->last_eob_len = ltree[END_BLOCK].Len; +} + +/* =========================================================================== + * Check if the data type is TEXT or BINARY, using the following algorithm: + * - TEXT if the two conditions below are satisfied: + * a) There are no non-portable control characters belonging to the + * "black list" (0..6, 14..25, 28..31). + * b) There is at least one printable character belonging to the + * "white list" (9 {TAB}, 10 {LF}, 13 {CR}, 32..255). + * - BINARY otherwise. + * - The following partially-portable control characters form a + * "gray list" that is ignored in this detection algorithm: + * (7 {BEL}, 8 {BS}, 11 {VT}, 12 {FF}, 26 {SUB}, 27 {ESC}). + * IN assertion: the fields Freq of dyn_ltree are set. + */ +local int detect_data_type(s) + deflate_state *s; +{ + /* black_mask is the bit mask of black-listed bytes + * set bits 0..6, 14..25, and 28..31 + * 0xf3ffc07f = binary 11110011111111111100000001111111 + */ + unsigned long black_mask = 0xf3ffc07fUL; + int n; + + /* Check for non-textual ("black-listed") bytes. */ + for (n = 0; n <= 31; n++, black_mask >>= 1) + if ((black_mask & 1) && (s->dyn_ltree[n].Freq != 0)) + return Z_BINARY; + + /* Check for textual ("white-listed") bytes. */ + if (s->dyn_ltree[9].Freq != 0 || s->dyn_ltree[10].Freq != 0 + || s->dyn_ltree[13].Freq != 0) + return Z_TEXT; + for (n = 32; n < LITERALS; n++) + if (s->dyn_ltree[n].Freq != 0) + return Z_TEXT; + + /* There are no "black-listed" or "white-listed" bytes: + * this stream either is empty or has tolerated ("gray-listed") bytes only. + */ + return Z_BINARY; +} + +/* =========================================================================== + * Reverse the first len bits of a code, using straightforward code (a faster + * method would use a table) + * IN assertion: 1 <= len <= 15 + */ +local unsigned bi_reverse(code, len) + unsigned code; /* the value to invert */ + int len; /* its bit length */ +{ + register unsigned res = 0; + do { + res |= code & 1; + code >>= 1, res <<= 1; + } while (--len > 0); + return res >> 1; +} + +/* =========================================================================== + * Flush the bit buffer, keeping at most 7 bits in it. + */ +local void bi_flush(s) + deflate_state *s; +{ + if (s->bi_valid == 16) { + put_short(s, s->bi_buf); + s->bi_buf = 0; + s->bi_valid = 0; + } else if (s->bi_valid >= 8) { + put_byte(s, (Byte)s->bi_buf); + s->bi_buf >>= 8; + s->bi_valid -= 8; + } +} + +/* =========================================================================== + * Flush the bit buffer and align the output on a byte boundary + */ +local void bi_windup(s) + deflate_state *s; +{ + if (s->bi_valid > 8) { + put_short(s, s->bi_buf); + } else if (s->bi_valid > 0) { + put_byte(s, (Byte)s->bi_buf); + } + s->bi_buf = 0; + s->bi_valid = 0; +#ifdef DEBUG + s->bits_sent = (s->bits_sent+7) & ~7; +#endif +} + +/* =========================================================================== + * Copy a stored block, storing first the length and its + * one's complement if requested. + */ +local void copy_block(s, buf, len, header) + deflate_state *s; + charf *buf; /* the input data */ + unsigned len; /* its length */ + int header; /* true if block header must be written */ +{ + bi_windup(s); /* align on byte boundary */ + s->last_eob_len = 8; /* enough lookahead for inflate */ + + if (header) { + put_short(s, (ush)len); + put_short(s, (ush)~len); +#ifdef DEBUG + s->bits_sent += 2*16; +#endif + } +#ifdef DEBUG + s->bits_sent += (ulg)len<<3; +#endif + while (len--) { + put_byte(s, *buf++); + } +} diff --git a/deps/zlib/trees.h b/deps/zlib/trees.h new file mode 100644 index 000000000..d35639d82 --- /dev/null +++ b/deps/zlib/trees.h @@ -0,0 +1,128 @@ +/* header created automatically with -DGEN_TREES_H */ + +local const ct_data static_ltree[L_CODES+2] = { +{{ 12},{ 8}}, {{140},{ 8}}, {{ 76},{ 8}}, {{204},{ 8}}, {{ 44},{ 8}}, +{{172},{ 8}}, {{108},{ 8}}, {{236},{ 8}}, {{ 28},{ 8}}, {{156},{ 8}}, +{{ 92},{ 8}}, {{220},{ 8}}, {{ 60},{ 8}}, {{188},{ 8}}, {{124},{ 8}}, +{{252},{ 8}}, {{ 2},{ 8}}, {{130},{ 8}}, {{ 66},{ 8}}, {{194},{ 8}}, +{{ 34},{ 8}}, {{162},{ 8}}, {{ 98},{ 8}}, {{226},{ 8}}, {{ 18},{ 8}}, +{{146},{ 8}}, {{ 82},{ 8}}, {{210},{ 8}}, {{ 50},{ 8}}, {{178},{ 8}}, +{{114},{ 8}}, {{242},{ 8}}, {{ 10},{ 8}}, {{138},{ 8}}, {{ 74},{ 8}}, +{{202},{ 8}}, {{ 42},{ 8}}, {{170},{ 8}}, {{106},{ 8}}, {{234},{ 8}}, +{{ 26},{ 8}}, {{154},{ 8}}, {{ 90},{ 8}}, {{218},{ 8}}, {{ 58},{ 8}}, +{{186},{ 8}}, {{122},{ 8}}, {{250},{ 8}}, {{ 6},{ 8}}, {{134},{ 8}}, +{{ 70},{ 8}}, {{198},{ 8}}, {{ 38},{ 8}}, {{166},{ 8}}, {{102},{ 8}}, +{{230},{ 8}}, {{ 22},{ 8}}, {{150},{ 8}}, {{ 86},{ 8}}, {{214},{ 8}}, +{{ 54},{ 8}}, {{182},{ 8}}, {{118},{ 8}}, {{246},{ 8}}, {{ 14},{ 8}}, +{{142},{ 8}}, {{ 78},{ 8}}, {{206},{ 8}}, {{ 46},{ 8}}, {{174},{ 8}}, +{{110},{ 8}}, {{238},{ 8}}, {{ 30},{ 8}}, {{158},{ 8}}, {{ 94},{ 8}}, +{{222},{ 8}}, {{ 62},{ 8}}, {{190},{ 8}}, {{126},{ 8}}, {{254},{ 8}}, +{{ 1},{ 8}}, {{129},{ 8}}, {{ 65},{ 8}}, {{193},{ 8}}, {{ 33},{ 8}}, +{{161},{ 8}}, {{ 97},{ 8}}, {{225},{ 8}}, {{ 17},{ 8}}, {{145},{ 8}}, +{{ 81},{ 8}}, {{209},{ 8}}, {{ 49},{ 8}}, {{177},{ 8}}, {{113},{ 8}}, +{{241},{ 8}}, {{ 9},{ 8}}, {{137},{ 8}}, {{ 73},{ 8}}, {{201},{ 8}}, +{{ 41},{ 8}}, {{169},{ 8}}, {{105},{ 8}}, {{233},{ 8}}, {{ 25},{ 8}}, +{{153},{ 8}}, {{ 89},{ 8}}, {{217},{ 8}}, {{ 57},{ 8}}, {{185},{ 8}}, +{{121},{ 8}}, {{249},{ 8}}, {{ 5},{ 8}}, {{133},{ 8}}, {{ 69},{ 8}}, +{{197},{ 8}}, {{ 37},{ 8}}, {{165},{ 8}}, {{101},{ 8}}, {{229},{ 8}}, +{{ 21},{ 8}}, {{149},{ 8}}, {{ 85},{ 8}}, {{213},{ 8}}, {{ 53},{ 8}}, +{{181},{ 8}}, {{117},{ 8}}, {{245},{ 8}}, {{ 13},{ 8}}, {{141},{ 8}}, +{{ 77},{ 8}}, {{205},{ 8}}, {{ 45},{ 8}}, {{173},{ 8}}, {{109},{ 8}}, +{{237},{ 8}}, {{ 29},{ 8}}, {{157},{ 8}}, {{ 93},{ 8}}, {{221},{ 8}}, +{{ 61},{ 8}}, {{189},{ 8}}, {{125},{ 8}}, {{253},{ 8}}, {{ 19},{ 9}}, +{{275},{ 9}}, {{147},{ 9}}, {{403},{ 9}}, {{ 83},{ 9}}, {{339},{ 9}}, +{{211},{ 9}}, {{467},{ 9}}, {{ 51},{ 9}}, {{307},{ 9}}, {{179},{ 9}}, +{{435},{ 9}}, {{115},{ 9}}, {{371},{ 9}}, {{243},{ 9}}, {{499},{ 9}}, +{{ 11},{ 9}}, {{267},{ 9}}, {{139},{ 9}}, {{395},{ 9}}, {{ 75},{ 9}}, +{{331},{ 9}}, {{203},{ 9}}, {{459},{ 9}}, {{ 43},{ 9}}, {{299},{ 9}}, +{{171},{ 9}}, {{427},{ 9}}, {{107},{ 9}}, {{363},{ 9}}, {{235},{ 9}}, +{{491},{ 9}}, {{ 27},{ 9}}, {{283},{ 9}}, {{155},{ 9}}, {{411},{ 9}}, +{{ 91},{ 9}}, {{347},{ 9}}, {{219},{ 9}}, {{475},{ 9}}, {{ 59},{ 9}}, +{{315},{ 9}}, {{187},{ 9}}, {{443},{ 9}}, {{123},{ 9}}, {{379},{ 9}}, +{{251},{ 9}}, {{507},{ 9}}, {{ 7},{ 9}}, {{263},{ 9}}, {{135},{ 9}}, +{{391},{ 9}}, {{ 71},{ 9}}, {{327},{ 9}}, {{199},{ 9}}, {{455},{ 9}}, +{{ 39},{ 9}}, {{295},{ 9}}, {{167},{ 9}}, {{423},{ 9}}, {{103},{ 9}}, +{{359},{ 9}}, {{231},{ 9}}, {{487},{ 9}}, {{ 23},{ 9}}, {{279},{ 9}}, +{{151},{ 9}}, {{407},{ 9}}, {{ 87},{ 9}}, {{343},{ 9}}, {{215},{ 9}}, +{{471},{ 9}}, {{ 55},{ 9}}, {{311},{ 9}}, {{183},{ 9}}, {{439},{ 9}}, +{{119},{ 9}}, {{375},{ 9}}, {{247},{ 9}}, {{503},{ 9}}, {{ 15},{ 9}}, +{{271},{ 9}}, {{143},{ 9}}, {{399},{ 9}}, {{ 79},{ 9}}, {{335},{ 9}}, +{{207},{ 9}}, {{463},{ 9}}, {{ 47},{ 9}}, {{303},{ 9}}, {{175},{ 9}}, +{{431},{ 9}}, {{111},{ 9}}, {{367},{ 9}}, {{239},{ 9}}, {{495},{ 9}}, +{{ 31},{ 9}}, {{287},{ 9}}, {{159},{ 9}}, {{415},{ 9}}, {{ 95},{ 9}}, +{{351},{ 9}}, {{223},{ 9}}, {{479},{ 9}}, {{ 63},{ 9}}, {{319},{ 9}}, +{{191},{ 9}}, {{447},{ 9}}, {{127},{ 9}}, {{383},{ 9}}, {{255},{ 9}}, +{{511},{ 9}}, {{ 0},{ 7}}, {{ 64},{ 7}}, {{ 32},{ 7}}, {{ 96},{ 7}}, +{{ 16},{ 7}}, {{ 80},{ 7}}, {{ 48},{ 7}}, {{112},{ 7}}, {{ 8},{ 7}}, +{{ 72},{ 7}}, {{ 40},{ 7}}, {{104},{ 7}}, {{ 24},{ 7}}, {{ 88},{ 7}}, +{{ 56},{ 7}}, {{120},{ 7}}, {{ 4},{ 7}}, {{ 68},{ 7}}, {{ 36},{ 7}}, +{{100},{ 7}}, {{ 20},{ 7}}, {{ 84},{ 7}}, {{ 52},{ 7}}, {{116},{ 7}}, +{{ 3},{ 8}}, {{131},{ 8}}, {{ 67},{ 8}}, {{195},{ 8}}, {{ 35},{ 8}}, +{{163},{ 8}}, {{ 99},{ 8}}, {{227},{ 8}} +}; + +local const ct_data static_dtree[D_CODES] = { +{{ 0},{ 5}}, {{16},{ 5}}, {{ 8},{ 5}}, {{24},{ 5}}, {{ 4},{ 5}}, +{{20},{ 5}}, {{12},{ 5}}, {{28},{ 5}}, {{ 2},{ 5}}, {{18},{ 5}}, +{{10},{ 5}}, {{26},{ 5}}, {{ 6},{ 5}}, {{22},{ 5}}, {{14},{ 5}}, +{{30},{ 5}}, {{ 1},{ 5}}, {{17},{ 5}}, {{ 9},{ 5}}, {{25},{ 5}}, +{{ 5},{ 5}}, {{21},{ 5}}, {{13},{ 5}}, {{29},{ 5}}, {{ 3},{ 5}}, +{{19},{ 5}}, {{11},{ 5}}, {{27},{ 5}}, {{ 7},{ 5}}, {{23},{ 5}} +}; + +const uch ZLIB_INTERNAL _dist_code[DIST_CODE_LEN] = { + 0, 1, 2, 3, 4, 4, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7, 8, 8, 8, 8, + 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10, +10, 10, 10, 10, 10, 10, 10, 10, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, +11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, +12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, +13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, +13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, +14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, +14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, +14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15, +15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, +15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, +15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 0, 0, 16, 17, +18, 18, 19, 19, 20, 20, 20, 20, 21, 21, 21, 21, 22, 22, 22, 22, 22, 22, 22, 22, +23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, +24, 24, 24, 24, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, +26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, +26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, +27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, +27, 27, 27, 27, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, +28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, +28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, 28, +28, 28, 28, 28, 28, 28, 28, 28, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, +29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, +29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, +29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29, 29 +}; + +const uch ZLIB_INTERNAL _length_code[MAX_MATCH-MIN_MATCH+1]= { + 0, 1, 2, 3, 4, 5, 6, 7, 8, 8, 9, 9, 10, 10, 11, 11, 12, 12, 12, 12, +13, 13, 13, 13, 14, 14, 14, 14, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, +17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 18, 18, 18, 18, 18, 18, 19, 19, 19, 19, +19, 19, 19, 19, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, +21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 21, 22, 22, 22, 22, +22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 22, 23, 23, 23, 23, 23, 23, 23, 23, +23, 23, 23, 23, 23, 23, 23, 23, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, +24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, +25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, +25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 25, 26, 26, 26, 26, 26, 26, 26, 26, +26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, 26, +26, 26, 26, 26, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, +27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 28 +}; + +local const int base_length[LENGTH_CODES] = { +0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 14, 16, 20, 24, 28, 32, 40, 48, 56, +64, 80, 96, 112, 128, 160, 192, 224, 0 +}; + +local const int base_dist[D_CODES] = { + 0, 1, 2, 3, 4, 6, 8, 12, 16, 24, + 32, 48, 64, 96, 128, 192, 256, 384, 512, 768, + 1024, 1536, 2048, 3072, 4096, 6144, 8192, 12288, 16384, 24576 +}; + diff --git a/deps/zlib/zconf.h b/deps/zlib/zconf.h new file mode 100644 index 000000000..b23438744 --- /dev/null +++ b/deps/zlib/zconf.h @@ -0,0 +1,428 @@ +/* zconf.h -- configuration of the zlib compression library + * Copyright (C) 1995-2010 Jean-loup Gailly. + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* @(#) $Id$ */ + +#ifndef ZCONF_H +#define ZCONF_H + +/* + * If you *really* need a unique prefix for all types and library functions, + * compile with -DZ_PREFIX. The "standard" zlib should be compiled without it. + * Even better than compiling with -DZ_PREFIX would be to use configure to set + * this permanently in zconf.h using "./configure --zprefix". + */ +#ifdef Z_PREFIX /* may be set to #if 1 by ./configure */ + +/* all linked symbols */ +# define _dist_code z__dist_code +# define _length_code z__length_code +# define _tr_align z__tr_align +# define _tr_flush_block z__tr_flush_block +# define _tr_init z__tr_init +# define _tr_stored_block z__tr_stored_block +# define _tr_tally z__tr_tally +# define adler32 z_adler32 +# define adler32_combine z_adler32_combine +# define adler32_combine64 z_adler32_combine64 +# define compress z_compress +# define compress2 z_compress2 +# define compressBound z_compressBound +# define crc32 z_crc32 +# define crc32_combine z_crc32_combine +# define crc32_combine64 z_crc32_combine64 +# define deflate z_deflate +# define deflateBound z_deflateBound +# define deflateCopy z_deflateCopy +# define deflateEnd z_deflateEnd +# define deflateInit2_ z_deflateInit2_ +# define deflateInit_ z_deflateInit_ +# define deflateParams z_deflateParams +# define deflatePrime z_deflatePrime +# define deflateReset z_deflateReset +# define deflateSetDictionary z_deflateSetDictionary +# define deflateSetHeader z_deflateSetHeader +# define deflateTune z_deflateTune +# define deflate_copyright z_deflate_copyright +# define get_crc_table z_get_crc_table +# define gz_error z_gz_error +# define gz_intmax z_gz_intmax +# define gz_strwinerror z_gz_strwinerror +# define gzbuffer z_gzbuffer +# define gzclearerr z_gzclearerr +# define gzclose z_gzclose +# define gzclose_r z_gzclose_r +# define gzclose_w z_gzclose_w +# define gzdirect z_gzdirect +# define gzdopen z_gzdopen +# define gzeof z_gzeof +# define gzerror z_gzerror +# define gzflush z_gzflush +# define gzgetc z_gzgetc +# define gzgets z_gzgets +# define gzoffset z_gzoffset +# define gzoffset64 z_gzoffset64 +# define gzopen z_gzopen +# define gzopen64 z_gzopen64 +# define gzprintf z_gzprintf +# define gzputc z_gzputc +# define gzputs z_gzputs +# define gzread z_gzread +# define gzrewind z_gzrewind +# define gzseek z_gzseek +# define gzseek64 z_gzseek64 +# define gzsetparams z_gzsetparams +# define gztell z_gztell +# define gztell64 z_gztell64 +# define gzungetc z_gzungetc +# define gzwrite z_gzwrite +# define inflate z_inflate +# define inflateBack z_inflateBack +# define inflateBackEnd z_inflateBackEnd +# define inflateBackInit_ z_inflateBackInit_ +# define inflateCopy z_inflateCopy +# define inflateEnd z_inflateEnd +# define inflateGetHeader z_inflateGetHeader +# define inflateInit2_ z_inflateInit2_ +# define inflateInit_ z_inflateInit_ +# define inflateMark z_inflateMark +# define inflatePrime z_inflatePrime +# define inflateReset z_inflateReset +# define inflateReset2 z_inflateReset2 +# define inflateSetDictionary z_inflateSetDictionary +# define inflateSync z_inflateSync +# define inflateSyncPoint z_inflateSyncPoint +# define inflateUndermine z_inflateUndermine +# define inflate_copyright z_inflate_copyright +# define inflate_fast z_inflate_fast +# define inflate_table z_inflate_table +# define uncompress z_uncompress +# define zError z_zError +# define zcalloc z_zcalloc +# define zcfree z_zcfree +# define zlibCompileFlags z_zlibCompileFlags +# define zlibVersion z_zlibVersion + +/* all zlib typedefs in zlib.h and zconf.h */ +# define Byte z_Byte +# define Bytef z_Bytef +# define alloc_func z_alloc_func +# define charf z_charf +# define free_func z_free_func +# define gzFile z_gzFile +# define gz_header z_gz_header +# define gz_headerp z_gz_headerp +# define in_func z_in_func +# define intf z_intf +# define out_func z_out_func +# define uInt z_uInt +# define uIntf z_uIntf +# define uLong z_uLong +# define uLongf z_uLongf +# define voidp z_voidp +# define voidpc z_voidpc +# define voidpf z_voidpf + +/* all zlib structs in zlib.h and zconf.h */ +# define gz_header_s z_gz_header_s +# define internal_state z_internal_state + +#endif + +#if defined(__MSDOS__) && !defined(MSDOS) +# define MSDOS +#endif +#if (defined(OS_2) || defined(__OS2__)) && !defined(OS2) +# define OS2 +#endif +#if defined(_WINDOWS) && !defined(WINDOWS) +# define WINDOWS +#endif +#if defined(_WIN32) || defined(_WIN32_WCE) || defined(__WIN32__) +# ifndef WIN32 +# define WIN32 +# endif +#endif +#if (defined(MSDOS) || defined(OS2) || defined(WINDOWS)) && !defined(WIN32) +# if !defined(__GNUC__) && !defined(__FLAT__) && !defined(__386__) +# ifndef SYS16BIT +# define SYS16BIT +# endif +# endif +#endif + +/* + * Compile with -DMAXSEG_64K if the alloc function cannot allocate more + * than 64k bytes at a time (needed on systems with 16-bit int). + */ +#ifdef SYS16BIT +# define MAXSEG_64K +#endif +#ifdef MSDOS +# define UNALIGNED_OK +#endif + +#ifdef __STDC_VERSION__ +# ifndef STDC +# define STDC +# endif +# if __STDC_VERSION__ >= 199901L +# ifndef STDC99 +# define STDC99 +# endif +# endif +#endif +#if !defined(STDC) && (defined(__STDC__) || defined(__cplusplus)) +# define STDC +#endif +#if !defined(STDC) && (defined(__GNUC__) || defined(__BORLANDC__)) +# define STDC +#endif +#if !defined(STDC) && (defined(MSDOS) || defined(WINDOWS) || defined(WIN32)) +# define STDC +#endif +#if !defined(STDC) && (defined(OS2) || defined(__HOS_AIX__)) +# define STDC +#endif + +#if defined(__OS400__) && !defined(STDC) /* iSeries (formerly AS/400). */ +# define STDC +#endif + +#ifndef STDC +# ifndef const /* cannot use !defined(STDC) && !defined(const) on Mac */ +# define const /* note: need a more gentle solution here */ +# endif +#endif + +/* Some Mac compilers merge all .h files incorrectly: */ +#if defined(__MWERKS__)||defined(applec)||defined(THINK_C)||defined(__SC__) +# define NO_DUMMY_DECL +#endif + +/* Maximum value for memLevel in deflateInit2 */ +#ifndef MAX_MEM_LEVEL +# ifdef MAXSEG_64K +# define MAX_MEM_LEVEL 8 +# else +# define MAX_MEM_LEVEL 9 +# endif +#endif + +/* Maximum value for windowBits in deflateInit2 and inflateInit2. + * WARNING: reducing MAX_WBITS makes minigzip unable to extract .gz files + * created by gzip. (Files created by minigzip can still be extracted by + * gzip.) + */ +#ifndef MAX_WBITS +# define MAX_WBITS 15 /* 32K LZ77 window */ +#endif + +/* The memory requirements for deflate are (in bytes): + (1 << (windowBits+2)) + (1 << (memLevel+9)) + that is: 128K for windowBits=15 + 128K for memLevel = 8 (default values) + plus a few kilobytes for small objects. For example, if you want to reduce + the default memory requirements from 256K to 128K, compile with + make CFLAGS="-O -DMAX_WBITS=14 -DMAX_MEM_LEVEL=7" + Of course this will generally degrade compression (there's no free lunch). + + The memory requirements for inflate are (in bytes) 1 << windowBits + that is, 32K for windowBits=15 (default value) plus a few kilobytes + for small objects. +*/ + + /* Type declarations */ + +#ifndef OF /* function prototypes */ +# ifdef STDC +# define OF(args) args +# else +# define OF(args) () +# endif +#endif + +/* The following definitions for FAR are needed only for MSDOS mixed + * model programming (small or medium model with some far allocations). + * This was tested only with MSC; for other MSDOS compilers you may have + * to define NO_MEMCPY in zutil.h. If you don't need the mixed model, + * just define FAR to be empty. + */ +#ifdef SYS16BIT +# if defined(M_I86SM) || defined(M_I86MM) + /* MSC small or medium model */ +# define SMALL_MEDIUM +# ifdef _MSC_VER +# define FAR _far +# else +# define FAR far +# endif +# endif +# if (defined(__SMALL__) || defined(__MEDIUM__)) + /* Turbo C small or medium model */ +# define SMALL_MEDIUM +# ifdef __BORLANDC__ +# define FAR _far +# else +# define FAR far +# endif +# endif +#endif + +#if defined(WINDOWS) || defined(WIN32) + /* If building or using zlib as a DLL, define ZLIB_DLL. + * This is not mandatory, but it offers a little performance increase. + */ +# ifdef ZLIB_DLL +# if defined(WIN32) && (!defined(__BORLANDC__) || (__BORLANDC__ >= 0x500)) +# ifdef ZLIB_INTERNAL +# define ZEXTERN extern __declspec(dllexport) +# else +# define ZEXTERN extern __declspec(dllimport) +# endif +# endif +# endif /* ZLIB_DLL */ + /* If building or using zlib with the WINAPI/WINAPIV calling convention, + * define ZLIB_WINAPI. + * Caution: the standard ZLIB1.DLL is NOT compiled using ZLIB_WINAPI. + */ +# ifdef ZLIB_WINAPI +# ifdef FAR +# undef FAR +# endif +# include + /* No need for _export, use ZLIB.DEF instead. */ + /* For complete Windows compatibility, use WINAPI, not __stdcall. */ +# define ZEXPORT WINAPI +# ifdef WIN32 +# define ZEXPORTVA WINAPIV +# else +# define ZEXPORTVA FAR CDECL +# endif +# endif +#endif + +#if defined (__BEOS__) +# ifdef ZLIB_DLL +# ifdef ZLIB_INTERNAL +# define ZEXPORT __declspec(dllexport) +# define ZEXPORTVA __declspec(dllexport) +# else +# define ZEXPORT __declspec(dllimport) +# define ZEXPORTVA __declspec(dllimport) +# endif +# endif +#endif + +#ifndef ZEXTERN +# define ZEXTERN extern +#endif +#ifndef ZEXPORT +# define ZEXPORT +#endif +#ifndef ZEXPORTVA +# define ZEXPORTVA +#endif + +#ifndef FAR +# define FAR +#endif + +#if !defined(__MACTYPES__) +typedef unsigned char Byte; /* 8 bits */ +#endif +typedef unsigned int uInt; /* 16 bits or more */ +typedef unsigned long uLong; /* 32 bits or more */ + +#ifdef SMALL_MEDIUM + /* Borland C/C++ and some old MSC versions ignore FAR inside typedef */ +# define Bytef Byte FAR +#else + typedef Byte FAR Bytef; +#endif +typedef char FAR charf; +typedef int FAR intf; +typedef uInt FAR uIntf; +typedef uLong FAR uLongf; + +#ifdef STDC + typedef void const *voidpc; + typedef void FAR *voidpf; + typedef void *voidp; +#else + typedef Byte const *voidpc; + typedef Byte FAR *voidpf; + typedef Byte *voidp; +#endif + +#if 1 /* was set to #if 1 by ./configure */ +# define Z_HAVE_UNISTD_H +#endif + +#ifdef STDC +# include /* for off_t */ +#endif + +/* a little trick to accommodate both "#define _LARGEFILE64_SOURCE" and + * "#define _LARGEFILE64_SOURCE 1" as requesting 64-bit operations, (even + * though the former does not conform to the LFS document), but considering + * both "#undef _LARGEFILE64_SOURCE" and "#define _LARGEFILE64_SOURCE 0" as + * equivalently requesting no 64-bit operations + */ +#if -_LARGEFILE64_SOURCE - -1 == 1 +# undef _LARGEFILE64_SOURCE +#endif + +#if defined(Z_HAVE_UNISTD_H) || defined(_LARGEFILE64_SOURCE) +# include /* for SEEK_* and off_t */ +# ifdef VMS +# include /* for off_t */ +# endif +# ifndef z_off_t +# define z_off_t off_t +# endif +#endif + +#ifndef SEEK_SET +# define SEEK_SET 0 /* Seek from beginning of file. */ +# define SEEK_CUR 1 /* Seek from current position. */ +# define SEEK_END 2 /* Set file pointer to EOF plus "offset" */ +#endif + +#ifndef z_off_t +# define z_off_t long +#endif + +#if defined(_LARGEFILE64_SOURCE) && _LFS64_LARGEFILE-0 +# define z_off64_t off64_t +#else +# define z_off64_t z_off_t +#endif + +#if defined(__OS400__) +# define NO_vsnprintf +#endif + +#if defined(__MVS__) +# define NO_vsnprintf +#endif + +/* MVS linker does not support external names larger than 8 bytes */ +#if defined(__MVS__) + #pragma map(deflateInit_,"DEIN") + #pragma map(deflateInit2_,"DEIN2") + #pragma map(deflateEnd,"DEEND") + #pragma map(deflateBound,"DEBND") + #pragma map(inflateInit_,"ININ") + #pragma map(inflateInit2_,"ININ2") + #pragma map(inflateEnd,"INEND") + #pragma map(inflateSync,"INSY") + #pragma map(inflateSetDictionary,"INSEDI") + #pragma map(compressBound,"CMBND") + #pragma map(inflate_table,"INTABL") + #pragma map(inflate_fast,"INFA") + #pragma map(inflate_copyright,"INCOPY") +#endif + +#endif /* ZCONF_H */ diff --git a/deps/zlib/zlib.h b/deps/zlib/zlib.h new file mode 100644 index 000000000..bfbba83e8 --- /dev/null +++ b/deps/zlib/zlib.h @@ -0,0 +1,1613 @@ +/* zlib.h -- interface of the 'zlib' general purpose compression library + version 1.2.5, April 19th, 2010 + + Copyright (C) 1995-2010 Jean-loup Gailly and Mark Adler + + This software is provided 'as-is', without any express or implied + warranty. In no event will the authors be held liable for any damages + arising from the use of this software. + + Permission is granted to anyone to use this software for any purpose, + including commercial applications, and to alter it and redistribute it + freely, subject to the following restrictions: + + 1. The origin of this software must not be misrepresented; you must not + claim that you wrote the original software. If you use this software + in a product, an acknowledgment in the product documentation would be + appreciated but is not required. + 2. Altered source versions must be plainly marked as such, and must not be + misrepresented as being the original software. + 3. This notice may not be removed or altered from any source distribution. + + Jean-loup Gailly Mark Adler + jloup@gzip.org madler@alumni.caltech.edu + + + The data format used by the zlib library is described by RFCs (Request for + Comments) 1950 to 1952 in the files http://www.ietf.org/rfc/rfc1950.txt + (zlib format), rfc1951.txt (deflate format) and rfc1952.txt (gzip format). +*/ + +#ifndef ZLIB_H +#define ZLIB_H + +#include "zconf.h" + +#ifdef __cplusplus +extern "C" { +#endif + +#define ZLIB_VERSION "1.2.5" +#define ZLIB_VERNUM 0x1250 +#define ZLIB_VER_MAJOR 1 +#define ZLIB_VER_MINOR 2 +#define ZLIB_VER_REVISION 5 +#define ZLIB_VER_SUBREVISION 0 + +/* + The 'zlib' compression library provides in-memory compression and + decompression functions, including integrity checks of the uncompressed data. + This version of the library supports only one compression method (deflation) + but other algorithms will be added later and will have the same stream + interface. + + Compression can be done in a single step if the buffers are large enough, + or can be done by repeated calls of the compression function. In the latter + case, the application must provide more input and/or consume the output + (providing more output space) before each call. + + The compressed data format used by default by the in-memory functions is + the zlib format, which is a zlib wrapper documented in RFC 1950, wrapped + around a deflate stream, which is itself documented in RFC 1951. + + The library also supports reading and writing files in gzip (.gz) format + with an interface similar to that of stdio using the functions that start + with "gz". The gzip format is different from the zlib format. gzip is a + gzip wrapper, documented in RFC 1952, wrapped around a deflate stream. + + This library can optionally read and write gzip streams in memory as well. + + The zlib format was designed to be compact and fast for use in memory + and on communications channels. The gzip format was designed for single- + file compression on file systems, has a larger header than zlib to maintain + directory information, and uses a different, slower check method than zlib. + + The library does not install any signal handler. The decoder checks + the consistency of the compressed data, so the library should never crash + even in case of corrupted input. +*/ + +typedef voidpf (*alloc_func) OF((voidpf opaque, uInt items, uInt size)); +typedef void (*free_func) OF((voidpf opaque, voidpf address)); + +struct internal_state; + +typedef struct z_stream_s { + Bytef *next_in; /* next input byte */ + uInt avail_in; /* number of bytes available at next_in */ + uLong total_in; /* total nb of input bytes read so far */ + + Bytef *next_out; /* next output byte should be put there */ + uInt avail_out; /* remaining free space at next_out */ + uLong total_out; /* total nb of bytes output so far */ + + char *msg; /* last error message, NULL if no error */ + struct internal_state FAR *state; /* not visible by applications */ + + alloc_func zalloc; /* used to allocate the internal state */ + free_func zfree; /* used to free the internal state */ + voidpf opaque; /* private data object passed to zalloc and zfree */ + + int data_type; /* best guess about the data type: binary or text */ + uLong adler; /* adler32 value of the uncompressed data */ + uLong reserved; /* reserved for future use */ +} z_stream; + +typedef z_stream FAR *z_streamp; + +/* + gzip header information passed to and from zlib routines. See RFC 1952 + for more details on the meanings of these fields. +*/ +typedef struct gz_header_s { + int text; /* true if compressed data believed to be text */ + uLong time; /* modification time */ + int xflags; /* extra flags (not used when writing a gzip file) */ + int os; /* operating system */ + Bytef *extra; /* pointer to extra field or Z_NULL if none */ + uInt extra_len; /* extra field length (valid if extra != Z_NULL) */ + uInt extra_max; /* space at extra (only when reading header) */ + Bytef *name; /* pointer to zero-terminated file name or Z_NULL */ + uInt name_max; /* space at name (only when reading header) */ + Bytef *comment; /* pointer to zero-terminated comment or Z_NULL */ + uInt comm_max; /* space at comment (only when reading header) */ + int hcrc; /* true if there was or will be a header crc */ + int done; /* true when done reading gzip header (not used + when writing a gzip file) */ +} gz_header; + +typedef gz_header FAR *gz_headerp; + +/* + The application must update next_in and avail_in when avail_in has dropped + to zero. It must update next_out and avail_out when avail_out has dropped + to zero. The application must initialize zalloc, zfree and opaque before + calling the init function. All other fields are set by the compression + library and must not be updated by the application. + + The opaque value provided by the application will be passed as the first + parameter for calls of zalloc and zfree. This can be useful for custom + memory management. The compression library attaches no meaning to the + opaque value. + + zalloc must return Z_NULL if there is not enough memory for the object. + If zlib is used in a multi-threaded application, zalloc and zfree must be + thread safe. + + On 16-bit systems, the functions zalloc and zfree must be able to allocate + exactly 65536 bytes, but will not be required to allocate more than this if + the symbol MAXSEG_64K is defined (see zconf.h). WARNING: On MSDOS, pointers + returned by zalloc for objects of exactly 65536 bytes *must* have their + offset normalized to zero. The default allocation function provided by this + library ensures this (see zutil.c). To reduce memory requirements and avoid + any allocation of 64K objects, at the expense of compression ratio, compile + the library with -DMAX_WBITS=14 (see zconf.h). + + The fields total_in and total_out can be used for statistics or progress + reports. After compression, total_in holds the total size of the + uncompressed data and may be saved for use in the decompressor (particularly + if the decompressor wants to decompress everything in a single step). +*/ + + /* constants */ + +#define Z_NO_FLUSH 0 +#define Z_PARTIAL_FLUSH 1 +#define Z_SYNC_FLUSH 2 +#define Z_FULL_FLUSH 3 +#define Z_FINISH 4 +#define Z_BLOCK 5 +#define Z_TREES 6 +/* Allowed flush values; see deflate() and inflate() below for details */ + +#define Z_OK 0 +#define Z_STREAM_END 1 +#define Z_NEED_DICT 2 +#define Z_ERRNO (-1) +#define Z_STREAM_ERROR (-2) +#define Z_DATA_ERROR (-3) +#define Z_MEM_ERROR (-4) +#define Z_BUF_ERROR (-5) +#define Z_VERSION_ERROR (-6) +/* Return codes for the compression/decompression functions. Negative values + * are errors, positive values are used for special but normal events. + */ + +#define Z_NO_COMPRESSION 0 +#define Z_BEST_SPEED 1 +#define Z_BEST_COMPRESSION 9 +#define Z_DEFAULT_COMPRESSION (-1) +/* compression levels */ + +#define Z_FILTERED 1 +#define Z_HUFFMAN_ONLY 2 +#define Z_RLE 3 +#define Z_FIXED 4 +#define Z_DEFAULT_STRATEGY 0 +/* compression strategy; see deflateInit2() below for details */ + +#define Z_BINARY 0 +#define Z_TEXT 1 +#define Z_ASCII Z_TEXT /* for compatibility with 1.2.2 and earlier */ +#define Z_UNKNOWN 2 +/* Possible values of the data_type field (though see inflate()) */ + +#define Z_DEFLATED 8 +/* The deflate compression method (the only one supported in this version) */ + +#define Z_NULL 0 /* for initializing zalloc, zfree, opaque */ + +#define zlib_version zlibVersion() +/* for compatibility with versions < 1.0.2 */ + + + /* basic functions */ + +ZEXTERN const char * ZEXPORT zlibVersion OF((void)); +/* The application can compare zlibVersion and ZLIB_VERSION for consistency. + If the first character differs, the library code actually used is not + compatible with the zlib.h header file used by the application. This check + is automatically made by deflateInit and inflateInit. + */ + +/* +ZEXTERN int ZEXPORT deflateInit OF((z_streamp strm, int level)); + + Initializes the internal stream state for compression. The fields + zalloc, zfree and opaque must be initialized before by the caller. If + zalloc and zfree are set to Z_NULL, deflateInit updates them to use default + allocation functions. + + The compression level must be Z_DEFAULT_COMPRESSION, or between 0 and 9: + 1 gives best speed, 9 gives best compression, 0 gives no compression at all + (the input data is simply copied a block at a time). Z_DEFAULT_COMPRESSION + requests a default compromise between speed and compression (currently + equivalent to level 6). + + deflateInit returns Z_OK if success, Z_MEM_ERROR if there was not enough + memory, Z_STREAM_ERROR if level is not a valid compression level, or + Z_VERSION_ERROR if the zlib library version (zlib_version) is incompatible + with the version assumed by the caller (ZLIB_VERSION). msg is set to null + if there is no error message. deflateInit does not perform any compression: + this will be done by deflate(). +*/ + + +ZEXTERN int ZEXPORT deflate OF((z_streamp strm, int flush)); +/* + deflate compresses as much data as possible, and stops when the input + buffer becomes empty or the output buffer becomes full. It may introduce + some output latency (reading input without producing any output) except when + forced to flush. + + The detailed semantics are as follows. deflate performs one or both of the + following actions: + + - Compress more input starting at next_in and update next_in and avail_in + accordingly. If not all input can be processed (because there is not + enough room in the output buffer), next_in and avail_in are updated and + processing will resume at this point for the next call of deflate(). + + - Provide more output starting at next_out and update next_out and avail_out + accordingly. This action is forced if the parameter flush is non zero. + Forcing flush frequently degrades the compression ratio, so this parameter + should be set only when necessary (in interactive applications). Some + output may be provided even if flush is not set. + + Before the call of deflate(), the application should ensure that at least + one of the actions is possible, by providing more input and/or consuming more + output, and updating avail_in or avail_out accordingly; avail_out should + never be zero before the call. The application can consume the compressed + output when it wants, for example when the output buffer is full (avail_out + == 0), or after each call of deflate(). If deflate returns Z_OK and with + zero avail_out, it must be called again after making room in the output + buffer because there might be more output pending. + + Normally the parameter flush is set to Z_NO_FLUSH, which allows deflate to + decide how much data to accumulate before producing output, in order to + maximize compression. + + If the parameter flush is set to Z_SYNC_FLUSH, all pending output is + flushed to the output buffer and the output is aligned on a byte boundary, so + that the decompressor can get all input data available so far. (In + particular avail_in is zero after the call if enough output space has been + provided before the call.) Flushing may degrade compression for some + compression algorithms and so it should be used only when necessary. This + completes the current deflate block and follows it with an empty stored block + that is three bits plus filler bits to the next byte, followed by four bytes + (00 00 ff ff). + + If flush is set to Z_PARTIAL_FLUSH, all pending output is flushed to the + output buffer, but the output is not aligned to a byte boundary. All of the + input data so far will be available to the decompressor, as for Z_SYNC_FLUSH. + This completes the current deflate block and follows it with an empty fixed + codes block that is 10 bits long. This assures that enough bytes are output + in order for the decompressor to finish the block before the empty fixed code + block. + + If flush is set to Z_BLOCK, a deflate block is completed and emitted, as + for Z_SYNC_FLUSH, but the output is not aligned on a byte boundary, and up to + seven bits of the current block are held to be written as the next byte after + the next deflate block is completed. In this case, the decompressor may not + be provided enough bits at this point in order to complete decompression of + the data provided so far to the compressor. It may need to wait for the next + block to be emitted. This is for advanced applications that need to control + the emission of deflate blocks. + + If flush is set to Z_FULL_FLUSH, all output is flushed as with + Z_SYNC_FLUSH, and the compression state is reset so that decompression can + restart from this point if previous compressed data has been damaged or if + random access is desired. Using Z_FULL_FLUSH too often can seriously degrade + compression. + + If deflate returns with avail_out == 0, this function must be called again + with the same value of the flush parameter and more output space (updated + avail_out), until the flush is complete (deflate returns with non-zero + avail_out). In the case of a Z_FULL_FLUSH or Z_SYNC_FLUSH, make sure that + avail_out is greater than six to avoid repeated flush markers due to + avail_out == 0 on return. + + If the parameter flush is set to Z_FINISH, pending input is processed, + pending output is flushed and deflate returns with Z_STREAM_END if there was + enough output space; if deflate returns with Z_OK, this function must be + called again with Z_FINISH and more output space (updated avail_out) but no + more input data, until it returns with Z_STREAM_END or an error. After + deflate has returned Z_STREAM_END, the only possible operations on the stream + are deflateReset or deflateEnd. + + Z_FINISH can be used immediately after deflateInit if all the compression + is to be done in a single step. In this case, avail_out must be at least the + value returned by deflateBound (see below). If deflate does not return + Z_STREAM_END, then it must be called again as described above. + + deflate() sets strm->adler to the adler32 checksum of all input read + so far (that is, total_in bytes). + + deflate() may update strm->data_type if it can make a good guess about + the input data type (Z_BINARY or Z_TEXT). In doubt, the data is considered + binary. This field is only for information purposes and does not affect the + compression algorithm in any manner. + + deflate() returns Z_OK if some progress has been made (more input + processed or more output produced), Z_STREAM_END if all input has been + consumed and all output has been produced (only when flush is set to + Z_FINISH), Z_STREAM_ERROR if the stream state was inconsistent (for example + if next_in or next_out was Z_NULL), Z_BUF_ERROR if no progress is possible + (for example avail_in or avail_out was zero). Note that Z_BUF_ERROR is not + fatal, and deflate() can be called again with more input and more output + space to continue compressing. +*/ + + +ZEXTERN int ZEXPORT deflateEnd OF((z_streamp strm)); +/* + All dynamically allocated data structures for this stream are freed. + This function discards any unprocessed input and does not flush any pending + output. + + deflateEnd returns Z_OK if success, Z_STREAM_ERROR if the + stream state was inconsistent, Z_DATA_ERROR if the stream was freed + prematurely (some input or output was discarded). In the error case, msg + may be set but then points to a static string (which must not be + deallocated). +*/ + + +/* +ZEXTERN int ZEXPORT inflateInit OF((z_streamp strm)); + + Initializes the internal stream state for decompression. The fields + next_in, avail_in, zalloc, zfree and opaque must be initialized before by + the caller. If next_in is not Z_NULL and avail_in is large enough (the + exact value depends on the compression method), inflateInit determines the + compression method from the zlib header and allocates all data structures + accordingly; otherwise the allocation will be deferred to the first call of + inflate. If zalloc and zfree are set to Z_NULL, inflateInit updates them to + use default allocation functions. + + inflateInit returns Z_OK if success, Z_MEM_ERROR if there was not enough + memory, Z_VERSION_ERROR if the zlib library version is incompatible with the + version assumed by the caller, or Z_STREAM_ERROR if the parameters are + invalid, such as a null pointer to the structure. msg is set to null if + there is no error message. inflateInit does not perform any decompression + apart from possibly reading the zlib header if present: actual decompression + will be done by inflate(). (So next_in and avail_in may be modified, but + next_out and avail_out are unused and unchanged.) The current implementation + of inflateInit() does not process any header information -- that is deferred + until inflate() is called. +*/ + + +ZEXTERN int ZEXPORT inflate OF((z_streamp strm, int flush)); +/* + inflate decompresses as much data as possible, and stops when the input + buffer becomes empty or the output buffer becomes full. It may introduce + some output latency (reading input without producing any output) except when + forced to flush. + + The detailed semantics are as follows. inflate performs one or both of the + following actions: + + - Decompress more input starting at next_in and update next_in and avail_in + accordingly. If not all input can be processed (because there is not + enough room in the output buffer), next_in is updated and processing will + resume at this point for the next call of inflate(). + + - Provide more output starting at next_out and update next_out and avail_out + accordingly. inflate() provides as much output as possible, until there is + no more input data or no more space in the output buffer (see below about + the flush parameter). + + Before the call of inflate(), the application should ensure that at least + one of the actions is possible, by providing more input and/or consuming more + output, and updating the next_* and avail_* values accordingly. The + application can consume the uncompressed output when it wants, for example + when the output buffer is full (avail_out == 0), or after each call of + inflate(). If inflate returns Z_OK and with zero avail_out, it must be + called again after making room in the output buffer because there might be + more output pending. + + The flush parameter of inflate() can be Z_NO_FLUSH, Z_SYNC_FLUSH, Z_FINISH, + Z_BLOCK, or Z_TREES. Z_SYNC_FLUSH requests that inflate() flush as much + output as possible to the output buffer. Z_BLOCK requests that inflate() + stop if and when it gets to the next deflate block boundary. When decoding + the zlib or gzip format, this will cause inflate() to return immediately + after the header and before the first block. When doing a raw inflate, + inflate() will go ahead and process the first block, and will return when it + gets to the end of that block, or when it runs out of data. + + The Z_BLOCK option assists in appending to or combining deflate streams. + Also to assist in this, on return inflate() will set strm->data_type to the + number of unused bits in the last byte taken from strm->next_in, plus 64 if + inflate() is currently decoding the last block in the deflate stream, plus + 128 if inflate() returned immediately after decoding an end-of-block code or + decoding the complete header up to just before the first byte of the deflate + stream. The end-of-block will not be indicated until all of the uncompressed + data from that block has been written to strm->next_out. The number of + unused bits may in general be greater than seven, except when bit 7 of + data_type is set, in which case the number of unused bits will be less than + eight. data_type is set as noted here every time inflate() returns for all + flush options, and so can be used to determine the amount of currently + consumed input in bits. + + The Z_TREES option behaves as Z_BLOCK does, but it also returns when the + end of each deflate block header is reached, before any actual data in that + block is decoded. This allows the caller to determine the length of the + deflate block header for later use in random access within a deflate block. + 256 is added to the value of strm->data_type when inflate() returns + immediately after reaching the end of the deflate block header. + + inflate() should normally be called until it returns Z_STREAM_END or an + error. However if all decompression is to be performed in a single step (a + single call of inflate), the parameter flush should be set to Z_FINISH. In + this case all pending input is processed and all pending output is flushed; + avail_out must be large enough to hold all the uncompressed data. (The size + of the uncompressed data may have been saved by the compressor for this + purpose.) The next operation on this stream must be inflateEnd to deallocate + the decompression state. The use of Z_FINISH is never required, but can be + used to inform inflate that a faster approach may be used for the single + inflate() call. + + In this implementation, inflate() always flushes as much output as + possible to the output buffer, and always uses the faster approach on the + first call. So the only effect of the flush parameter in this implementation + is on the return value of inflate(), as noted below, or when it returns early + because Z_BLOCK or Z_TREES is used. + + If a preset dictionary is needed after this call (see inflateSetDictionary + below), inflate sets strm->adler to the adler32 checksum of the dictionary + chosen by the compressor and returns Z_NEED_DICT; otherwise it sets + strm->adler to the adler32 checksum of all output produced so far (that is, + total_out bytes) and returns Z_OK, Z_STREAM_END or an error code as described + below. At the end of the stream, inflate() checks that its computed adler32 + checksum is equal to that saved by the compressor and returns Z_STREAM_END + only if the checksum is correct. + + inflate() can decompress and check either zlib-wrapped or gzip-wrapped + deflate data. The header type is detected automatically, if requested when + initializing with inflateInit2(). Any information contained in the gzip + header is not retained, so applications that need that information should + instead use raw inflate, see inflateInit2() below, or inflateBack() and + perform their own processing of the gzip header and trailer. + + inflate() returns Z_OK if some progress has been made (more input processed + or more output produced), Z_STREAM_END if the end of the compressed data has + been reached and all uncompressed output has been produced, Z_NEED_DICT if a + preset dictionary is needed at this point, Z_DATA_ERROR if the input data was + corrupted (input stream not conforming to the zlib format or incorrect check + value), Z_STREAM_ERROR if the stream structure was inconsistent (for example + next_in or next_out was Z_NULL), Z_MEM_ERROR if there was not enough memory, + Z_BUF_ERROR if no progress is possible or if there was not enough room in the + output buffer when Z_FINISH is used. Note that Z_BUF_ERROR is not fatal, and + inflate() can be called again with more input and more output space to + continue decompressing. If Z_DATA_ERROR is returned, the application may + then call inflateSync() to look for a good compression block if a partial + recovery of the data is desired. +*/ + + +ZEXTERN int ZEXPORT inflateEnd OF((z_streamp strm)); +/* + All dynamically allocated data structures for this stream are freed. + This function discards any unprocessed input and does not flush any pending + output. + + inflateEnd returns Z_OK if success, Z_STREAM_ERROR if the stream state + was inconsistent. In the error case, msg may be set but then points to a + static string (which must not be deallocated). +*/ + + + /* Advanced functions */ + +/* + The following functions are needed only in some special applications. +*/ + +/* +ZEXTERN int ZEXPORT deflateInit2 OF((z_streamp strm, + int level, + int method, + int windowBits, + int memLevel, + int strategy)); + + This is another version of deflateInit with more compression options. The + fields next_in, zalloc, zfree and opaque must be initialized before by the + caller. + + The method parameter is the compression method. It must be Z_DEFLATED in + this version of the library. + + The windowBits parameter is the base two logarithm of the window size + (the size of the history buffer). It should be in the range 8..15 for this + version of the library. Larger values of this parameter result in better + compression at the expense of memory usage. The default value is 15 if + deflateInit is used instead. + + windowBits can also be -8..-15 for raw deflate. In this case, -windowBits + determines the window size. deflate() will then generate raw deflate data + with no zlib header or trailer, and will not compute an adler32 check value. + + windowBits can also be greater than 15 for optional gzip encoding. Add + 16 to windowBits to write a simple gzip header and trailer around the + compressed data instead of a zlib wrapper. The gzip header will have no + file name, no extra data, no comment, no modification time (set to zero), no + header crc, and the operating system will be set to 255 (unknown). If a + gzip stream is being written, strm->adler is a crc32 instead of an adler32. + + The memLevel parameter specifies how much memory should be allocated + for the internal compression state. memLevel=1 uses minimum memory but is + slow and reduces compression ratio; memLevel=9 uses maximum memory for + optimal speed. The default value is 8. See zconf.h for total memory usage + as a function of windowBits and memLevel. + + The strategy parameter is used to tune the compression algorithm. Use the + value Z_DEFAULT_STRATEGY for normal data, Z_FILTERED for data produced by a + filter (or predictor), Z_HUFFMAN_ONLY to force Huffman encoding only (no + string match), or Z_RLE to limit match distances to one (run-length + encoding). Filtered data consists mostly of small values with a somewhat + random distribution. In this case, the compression algorithm is tuned to + compress them better. The effect of Z_FILTERED is to force more Huffman + coding and less string matching; it is somewhat intermediate between + Z_DEFAULT_STRATEGY and Z_HUFFMAN_ONLY. Z_RLE is designed to be almost as + fast as Z_HUFFMAN_ONLY, but give better compression for PNG image data. The + strategy parameter only affects the compression ratio but not the + correctness of the compressed output even if it is not set appropriately. + Z_FIXED prevents the use of dynamic Huffman codes, allowing for a simpler + decoder for special applications. + + deflateInit2 returns Z_OK if success, Z_MEM_ERROR if there was not enough + memory, Z_STREAM_ERROR if any parameter is invalid (such as an invalid + method), or Z_VERSION_ERROR if the zlib library version (zlib_version) is + incompatible with the version assumed by the caller (ZLIB_VERSION). msg is + set to null if there is no error message. deflateInit2 does not perform any + compression: this will be done by deflate(). +*/ + +ZEXTERN int ZEXPORT deflateSetDictionary OF((z_streamp strm, + const Bytef *dictionary, + uInt dictLength)); +/* + Initializes the compression dictionary from the given byte sequence + without producing any compressed output. This function must be called + immediately after deflateInit, deflateInit2 or deflateReset, before any call + of deflate. The compressor and decompressor must use exactly the same + dictionary (see inflateSetDictionary). + + The dictionary should consist of strings (byte sequences) that are likely + to be encountered later in the data to be compressed, with the most commonly + used strings preferably put towards the end of the dictionary. Using a + dictionary is most useful when the data to be compressed is short and can be + predicted with good accuracy; the data can then be compressed better than + with the default empty dictionary. + + Depending on the size of the compression data structures selected by + deflateInit or deflateInit2, a part of the dictionary may in effect be + discarded, for example if the dictionary is larger than the window size + provided in deflateInit or deflateInit2. Thus the strings most likely to be + useful should be put at the end of the dictionary, not at the front. In + addition, the current implementation of deflate will use at most the window + size minus 262 bytes of the provided dictionary. + + Upon return of this function, strm->adler is set to the adler32 value + of the dictionary; the decompressor may later use this value to determine + which dictionary has been used by the compressor. (The adler32 value + applies to the whole dictionary even if only a subset of the dictionary is + actually used by the compressor.) If a raw deflate was requested, then the + adler32 value is not computed and strm->adler is not set. + + deflateSetDictionary returns Z_OK if success, or Z_STREAM_ERROR if a + parameter is invalid (e.g. dictionary being Z_NULL) or the stream state is + inconsistent (for example if deflate has already been called for this stream + or if the compression method is bsort). deflateSetDictionary does not + perform any compression: this will be done by deflate(). +*/ + +ZEXTERN int ZEXPORT deflateCopy OF((z_streamp dest, + z_streamp source)); +/* + Sets the destination stream as a complete copy of the source stream. + + This function can be useful when several compression strategies will be + tried, for example when there are several ways of pre-processing the input + data with a filter. The streams that will be discarded should then be freed + by calling deflateEnd. Note that deflateCopy duplicates the internal + compression state which can be quite large, so this strategy is slow and can + consume lots of memory. + + deflateCopy returns Z_OK if success, Z_MEM_ERROR if there was not + enough memory, Z_STREAM_ERROR if the source stream state was inconsistent + (such as zalloc being Z_NULL). msg is left unchanged in both source and + destination. +*/ + +ZEXTERN int ZEXPORT deflateReset OF((z_streamp strm)); +/* + This function is equivalent to deflateEnd followed by deflateInit, + but does not free and reallocate all the internal compression state. The + stream will keep the same compression level and any other attributes that + may have been set by deflateInit2. + + deflateReset returns Z_OK if success, or Z_STREAM_ERROR if the source + stream state was inconsistent (such as zalloc or state being Z_NULL). +*/ + +ZEXTERN int ZEXPORT deflateParams OF((z_streamp strm, + int level, + int strategy)); +/* + Dynamically update the compression level and compression strategy. The + interpretation of level and strategy is as in deflateInit2. This can be + used to switch between compression and straight copy of the input data, or + to switch to a different kind of input data requiring a different strategy. + If the compression level is changed, the input available so far is + compressed with the old level (and may be flushed); the new level will take + effect only at the next call of deflate(). + + Before the call of deflateParams, the stream state must be set as for + a call of deflate(), since the currently available input may have to be + compressed and flushed. In particular, strm->avail_out must be non-zero. + + deflateParams returns Z_OK if success, Z_STREAM_ERROR if the source + stream state was inconsistent or if a parameter was invalid, Z_BUF_ERROR if + strm->avail_out was zero. +*/ + +ZEXTERN int ZEXPORT deflateTune OF((z_streamp strm, + int good_length, + int max_lazy, + int nice_length, + int max_chain)); +/* + Fine tune deflate's internal compression parameters. This should only be + used by someone who understands the algorithm used by zlib's deflate for + searching for the best matching string, and even then only by the most + fanatic optimizer trying to squeeze out the last compressed bit for their + specific input data. Read the deflate.c source code for the meaning of the + max_lazy, good_length, nice_length, and max_chain parameters. + + deflateTune() can be called after deflateInit() or deflateInit2(), and + returns Z_OK on success, or Z_STREAM_ERROR for an invalid deflate stream. + */ + +ZEXTERN uLong ZEXPORT deflateBound OF((z_streamp strm, + uLong sourceLen)); +/* + deflateBound() returns an upper bound on the compressed size after + deflation of sourceLen bytes. It must be called after deflateInit() or + deflateInit2(), and after deflateSetHeader(), if used. This would be used + to allocate an output buffer for deflation in a single pass, and so would be + called before deflate(). +*/ + +ZEXTERN int ZEXPORT deflatePrime OF((z_streamp strm, + int bits, + int value)); +/* + deflatePrime() inserts bits in the deflate output stream. The intent + is that this function is used to start off the deflate output with the bits + leftover from a previous deflate stream when appending to it. As such, this + function can only be used for raw deflate, and must be used before the first + deflate() call after a deflateInit2() or deflateReset(). bits must be less + than or equal to 16, and that many of the least significant bits of value + will be inserted in the output. + + deflatePrime returns Z_OK if success, or Z_STREAM_ERROR if the source + stream state was inconsistent. +*/ + +ZEXTERN int ZEXPORT deflateSetHeader OF((z_streamp strm, + gz_headerp head)); +/* + deflateSetHeader() provides gzip header information for when a gzip + stream is requested by deflateInit2(). deflateSetHeader() may be called + after deflateInit2() or deflateReset() and before the first call of + deflate(). The text, time, os, extra field, name, and comment information + in the provided gz_header structure are written to the gzip header (xflag is + ignored -- the extra flags are set according to the compression level). The + caller must assure that, if not Z_NULL, name and comment are terminated with + a zero byte, and that if extra is not Z_NULL, that extra_len bytes are + available there. If hcrc is true, a gzip header crc is included. Note that + the current versions of the command-line version of gzip (up through version + 1.3.x) do not support header crc's, and will report that it is a "multi-part + gzip file" and give up. + + If deflateSetHeader is not used, the default gzip header has text false, + the time set to zero, and os set to 255, with no extra, name, or comment + fields. The gzip header is returned to the default state by deflateReset(). + + deflateSetHeader returns Z_OK if success, or Z_STREAM_ERROR if the source + stream state was inconsistent. +*/ + +/* +ZEXTERN int ZEXPORT inflateInit2 OF((z_streamp strm, + int windowBits)); + + This is another version of inflateInit with an extra parameter. The + fields next_in, avail_in, zalloc, zfree and opaque must be initialized + before by the caller. + + The windowBits parameter is the base two logarithm of the maximum window + size (the size of the history buffer). It should be in the range 8..15 for + this version of the library. The default value is 15 if inflateInit is used + instead. windowBits must be greater than or equal to the windowBits value + provided to deflateInit2() while compressing, or it must be equal to 15 if + deflateInit2() was not used. If a compressed stream with a larger window + size is given as input, inflate() will return with the error code + Z_DATA_ERROR instead of trying to allocate a larger window. + + windowBits can also be zero to request that inflate use the window size in + the zlib header of the compressed stream. + + windowBits can also be -8..-15 for raw inflate. In this case, -windowBits + determines the window size. inflate() will then process raw deflate data, + not looking for a zlib or gzip header, not generating a check value, and not + looking for any check values for comparison at the end of the stream. This + is for use with other formats that use the deflate compressed data format + such as zip. Those formats provide their own check values. If a custom + format is developed using the raw deflate format for compressed data, it is + recommended that a check value such as an adler32 or a crc32 be applied to + the uncompressed data as is done in the zlib, gzip, and zip formats. For + most applications, the zlib format should be used as is. Note that comments + above on the use in deflateInit2() applies to the magnitude of windowBits. + + windowBits can also be greater than 15 for optional gzip decoding. Add + 32 to windowBits to enable zlib and gzip decoding with automatic header + detection, or add 16 to decode only the gzip format (the zlib format will + return a Z_DATA_ERROR). If a gzip stream is being decoded, strm->adler is a + crc32 instead of an adler32. + + inflateInit2 returns Z_OK if success, Z_MEM_ERROR if there was not enough + memory, Z_VERSION_ERROR if the zlib library version is incompatible with the + version assumed by the caller, or Z_STREAM_ERROR if the parameters are + invalid, such as a null pointer to the structure. msg is set to null if + there is no error message. inflateInit2 does not perform any decompression + apart from possibly reading the zlib header if present: actual decompression + will be done by inflate(). (So next_in and avail_in may be modified, but + next_out and avail_out are unused and unchanged.) The current implementation + of inflateInit2() does not process any header information -- that is + deferred until inflate() is called. +*/ + +ZEXTERN int ZEXPORT inflateSetDictionary OF((z_streamp strm, + const Bytef *dictionary, + uInt dictLength)); +/* + Initializes the decompression dictionary from the given uncompressed byte + sequence. This function must be called immediately after a call of inflate, + if that call returned Z_NEED_DICT. The dictionary chosen by the compressor + can be determined from the adler32 value returned by that call of inflate. + The compressor and decompressor must use exactly the same dictionary (see + deflateSetDictionary). For raw inflate, this function can be called + immediately after inflateInit2() or inflateReset() and before any call of + inflate() to set the dictionary. The application must insure that the + dictionary that was used for compression is provided. + + inflateSetDictionary returns Z_OK if success, Z_STREAM_ERROR if a + parameter is invalid (e.g. dictionary being Z_NULL) or the stream state is + inconsistent, Z_DATA_ERROR if the given dictionary doesn't match the + expected one (incorrect adler32 value). inflateSetDictionary does not + perform any decompression: this will be done by subsequent calls of + inflate(). +*/ + +ZEXTERN int ZEXPORT inflateSync OF((z_streamp strm)); +/* + Skips invalid compressed data until a full flush point (see above the + description of deflate with Z_FULL_FLUSH) can be found, or until all + available input is skipped. No output is provided. + + inflateSync returns Z_OK if a full flush point has been found, Z_BUF_ERROR + if no more input was provided, Z_DATA_ERROR if no flush point has been + found, or Z_STREAM_ERROR if the stream structure was inconsistent. In the + success case, the application may save the current current value of total_in + which indicates where valid compressed data was found. In the error case, + the application may repeatedly call inflateSync, providing more input each + time, until success or end of the input data. +*/ + +ZEXTERN int ZEXPORT inflateCopy OF((z_streamp dest, + z_streamp source)); +/* + Sets the destination stream as a complete copy of the source stream. + + This function can be useful when randomly accessing a large stream. The + first pass through the stream can periodically record the inflate state, + allowing restarting inflate at those points when randomly accessing the + stream. + + inflateCopy returns Z_OK if success, Z_MEM_ERROR if there was not + enough memory, Z_STREAM_ERROR if the source stream state was inconsistent + (such as zalloc being Z_NULL). msg is left unchanged in both source and + destination. +*/ + +ZEXTERN int ZEXPORT inflateReset OF((z_streamp strm)); +/* + This function is equivalent to inflateEnd followed by inflateInit, + but does not free and reallocate all the internal decompression state. The + stream will keep attributes that may have been set by inflateInit2. + + inflateReset returns Z_OK if success, or Z_STREAM_ERROR if the source + stream state was inconsistent (such as zalloc or state being Z_NULL). +*/ + +ZEXTERN int ZEXPORT inflateReset2 OF((z_streamp strm, + int windowBits)); +/* + This function is the same as inflateReset, but it also permits changing + the wrap and window size requests. The windowBits parameter is interpreted + the same as it is for inflateInit2. + + inflateReset2 returns Z_OK if success, or Z_STREAM_ERROR if the source + stream state was inconsistent (such as zalloc or state being Z_NULL), or if + the windowBits parameter is invalid. +*/ + +ZEXTERN int ZEXPORT inflatePrime OF((z_streamp strm, + int bits, + int value)); +/* + This function inserts bits in the inflate input stream. The intent is + that this function is used to start inflating at a bit position in the + middle of a byte. The provided bits will be used before any bytes are used + from next_in. This function should only be used with raw inflate, and + should be used before the first inflate() call after inflateInit2() or + inflateReset(). bits must be less than or equal to 16, and that many of the + least significant bits of value will be inserted in the input. + + If bits is negative, then the input stream bit buffer is emptied. Then + inflatePrime() can be called again to put bits in the buffer. This is used + to clear out bits leftover after feeding inflate a block description prior + to feeding inflate codes. + + inflatePrime returns Z_OK if success, or Z_STREAM_ERROR if the source + stream state was inconsistent. +*/ + +ZEXTERN long ZEXPORT inflateMark OF((z_streamp strm)); +/* + This function returns two values, one in the lower 16 bits of the return + value, and the other in the remaining upper bits, obtained by shifting the + return value down 16 bits. If the upper value is -1 and the lower value is + zero, then inflate() is currently decoding information outside of a block. + If the upper value is -1 and the lower value is non-zero, then inflate is in + the middle of a stored block, with the lower value equaling the number of + bytes from the input remaining to copy. If the upper value is not -1, then + it is the number of bits back from the current bit position in the input of + the code (literal or length/distance pair) currently being processed. In + that case the lower value is the number of bytes already emitted for that + code. + + A code is being processed if inflate is waiting for more input to complete + decoding of the code, or if it has completed decoding but is waiting for + more output space to write the literal or match data. + + inflateMark() is used to mark locations in the input data for random + access, which may be at bit positions, and to note those cases where the + output of a code may span boundaries of random access blocks. The current + location in the input stream can be determined from avail_in and data_type + as noted in the description for the Z_BLOCK flush parameter for inflate. + + inflateMark returns the value noted above or -1 << 16 if the provided + source stream state was inconsistent. +*/ + +ZEXTERN int ZEXPORT inflateGetHeader OF((z_streamp strm, + gz_headerp head)); +/* + inflateGetHeader() requests that gzip header information be stored in the + provided gz_header structure. inflateGetHeader() may be called after + inflateInit2() or inflateReset(), and before the first call of inflate(). + As inflate() processes the gzip stream, head->done is zero until the header + is completed, at which time head->done is set to one. If a zlib stream is + being decoded, then head->done is set to -1 to indicate that there will be + no gzip header information forthcoming. Note that Z_BLOCK or Z_TREES can be + used to force inflate() to return immediately after header processing is + complete and before any actual data is decompressed. + + The text, time, xflags, and os fields are filled in with the gzip header + contents. hcrc is set to true if there is a header CRC. (The header CRC + was valid if done is set to one.) If extra is not Z_NULL, then extra_max + contains the maximum number of bytes to write to extra. Once done is true, + extra_len contains the actual extra field length, and extra contains the + extra field, or that field truncated if extra_max is less than extra_len. + If name is not Z_NULL, then up to name_max characters are written there, + terminated with a zero unless the length is greater than name_max. If + comment is not Z_NULL, then up to comm_max characters are written there, + terminated with a zero unless the length is greater than comm_max. When any + of extra, name, or comment are not Z_NULL and the respective field is not + present in the header, then that field is set to Z_NULL to signal its + absence. This allows the use of deflateSetHeader() with the returned + structure to duplicate the header. However if those fields are set to + allocated memory, then the application will need to save those pointers + elsewhere so that they can be eventually freed. + + If inflateGetHeader is not used, then the header information is simply + discarded. The header is always checked for validity, including the header + CRC if present. inflateReset() will reset the process to discard the header + information. The application would need to call inflateGetHeader() again to + retrieve the header from the next gzip stream. + + inflateGetHeader returns Z_OK if success, or Z_STREAM_ERROR if the source + stream state was inconsistent. +*/ + +/* +ZEXTERN int ZEXPORT inflateBackInit OF((z_streamp strm, int windowBits, + unsigned char FAR *window)); + + Initialize the internal stream state for decompression using inflateBack() + calls. The fields zalloc, zfree and opaque in strm must be initialized + before the call. If zalloc and zfree are Z_NULL, then the default library- + derived memory allocation routines are used. windowBits is the base two + logarithm of the window size, in the range 8..15. window is a caller + supplied buffer of that size. Except for special applications where it is + assured that deflate was used with small window sizes, windowBits must be 15 + and a 32K byte window must be supplied to be able to decompress general + deflate streams. + + See inflateBack() for the usage of these routines. + + inflateBackInit will return Z_OK on success, Z_STREAM_ERROR if any of + the paramaters are invalid, Z_MEM_ERROR if the internal state could not be + allocated, or Z_VERSION_ERROR if the version of the library does not match + the version of the header file. +*/ + +typedef unsigned (*in_func) OF((void FAR *, unsigned char FAR * FAR *)); +typedef int (*out_func) OF((void FAR *, unsigned char FAR *, unsigned)); + +ZEXTERN int ZEXPORT inflateBack OF((z_streamp strm, + in_func in, void FAR *in_desc, + out_func out, void FAR *out_desc)); +/* + inflateBack() does a raw inflate with a single call using a call-back + interface for input and output. This is more efficient than inflate() for + file i/o applications in that it avoids copying between the output and the + sliding window by simply making the window itself the output buffer. This + function trusts the application to not change the output buffer passed by + the output function, at least until inflateBack() returns. + + inflateBackInit() must be called first to allocate the internal state + and to initialize the state with the user-provided window buffer. + inflateBack() may then be used multiple times to inflate a complete, raw + deflate stream with each call. inflateBackEnd() is then called to free the + allocated state. + + A raw deflate stream is one with no zlib or gzip header or trailer. + This routine would normally be used in a utility that reads zip or gzip + files and writes out uncompressed files. The utility would decode the + header and process the trailer on its own, hence this routine expects only + the raw deflate stream to decompress. This is different from the normal + behavior of inflate(), which expects either a zlib or gzip header and + trailer around the deflate stream. + + inflateBack() uses two subroutines supplied by the caller that are then + called by inflateBack() for input and output. inflateBack() calls those + routines until it reads a complete deflate stream and writes out all of the + uncompressed data, or until it encounters an error. The function's + parameters and return types are defined above in the in_func and out_func + typedefs. inflateBack() will call in(in_desc, &buf) which should return the + number of bytes of provided input, and a pointer to that input in buf. If + there is no input available, in() must return zero--buf is ignored in that + case--and inflateBack() will return a buffer error. inflateBack() will call + out(out_desc, buf, len) to write the uncompressed data buf[0..len-1]. out() + should return zero on success, or non-zero on failure. If out() returns + non-zero, inflateBack() will return with an error. Neither in() nor out() + are permitted to change the contents of the window provided to + inflateBackInit(), which is also the buffer that out() uses to write from. + The length written by out() will be at most the window size. Any non-zero + amount of input may be provided by in(). + + For convenience, inflateBack() can be provided input on the first call by + setting strm->next_in and strm->avail_in. If that input is exhausted, then + in() will be called. Therefore strm->next_in must be initialized before + calling inflateBack(). If strm->next_in is Z_NULL, then in() will be called + immediately for input. If strm->next_in is not Z_NULL, then strm->avail_in + must also be initialized, and then if strm->avail_in is not zero, input will + initially be taken from strm->next_in[0 .. strm->avail_in - 1]. + + The in_desc and out_desc parameters of inflateBack() is passed as the + first parameter of in() and out() respectively when they are called. These + descriptors can be optionally used to pass any information that the caller- + supplied in() and out() functions need to do their job. + + On return, inflateBack() will set strm->next_in and strm->avail_in to + pass back any unused input that was provided by the last in() call. The + return values of inflateBack() can be Z_STREAM_END on success, Z_BUF_ERROR + if in() or out() returned an error, Z_DATA_ERROR if there was a format error + in the deflate stream (in which case strm->msg is set to indicate the nature + of the error), or Z_STREAM_ERROR if the stream was not properly initialized. + In the case of Z_BUF_ERROR, an input or output error can be distinguished + using strm->next_in which will be Z_NULL only if in() returned an error. If + strm->next_in is not Z_NULL, then the Z_BUF_ERROR was due to out() returning + non-zero. (in() will always be called before out(), so strm->next_in is + assured to be defined if out() returns non-zero.) Note that inflateBack() + cannot return Z_OK. +*/ + +ZEXTERN int ZEXPORT inflateBackEnd OF((z_streamp strm)); +/* + All memory allocated by inflateBackInit() is freed. + + inflateBackEnd() returns Z_OK on success, or Z_STREAM_ERROR if the stream + state was inconsistent. +*/ + +ZEXTERN uLong ZEXPORT zlibCompileFlags OF((void)); +/* Return flags indicating compile-time options. + + Type sizes, two bits each, 00 = 16 bits, 01 = 32, 10 = 64, 11 = other: + 1.0: size of uInt + 3.2: size of uLong + 5.4: size of voidpf (pointer) + 7.6: size of z_off_t + + Compiler, assembler, and debug options: + 8: DEBUG + 9: ASMV or ASMINF -- use ASM code + 10: ZLIB_WINAPI -- exported functions use the WINAPI calling convention + 11: 0 (reserved) + + One-time table building (smaller code, but not thread-safe if true): + 12: BUILDFIXED -- build static block decoding tables when needed + 13: DYNAMIC_CRC_TABLE -- build CRC calculation tables when needed + 14,15: 0 (reserved) + + Library content (indicates missing functionality): + 16: NO_GZCOMPRESS -- gz* functions cannot compress (to avoid linking + deflate code when not needed) + 17: NO_GZIP -- deflate can't write gzip streams, and inflate can't detect + and decode gzip streams (to avoid linking crc code) + 18-19: 0 (reserved) + + Operation variations (changes in library functionality): + 20: PKZIP_BUG_WORKAROUND -- slightly more permissive inflate + 21: FASTEST -- deflate algorithm with only one, lowest compression level + 22,23: 0 (reserved) + + The sprintf variant used by gzprintf (zero is best): + 24: 0 = vs*, 1 = s* -- 1 means limited to 20 arguments after the format + 25: 0 = *nprintf, 1 = *printf -- 1 means gzprintf() not secure! + 26: 0 = returns value, 1 = void -- 1 means inferred string length returned + + Remainder: + 27-31: 0 (reserved) + */ + + + /* utility functions */ + +/* + The following utility functions are implemented on top of the basic + stream-oriented functions. To simplify the interface, some default options + are assumed (compression level and memory usage, standard memory allocation + functions). The source code of these utility functions can be modified if + you need special options. +*/ + +ZEXTERN int ZEXPORT compress OF((Bytef *dest, uLongf *destLen, + const Bytef *source, uLong sourceLen)); +/* + Compresses the source buffer into the destination buffer. sourceLen is + the byte length of the source buffer. Upon entry, destLen is the total size + of the destination buffer, which must be at least the value returned by + compressBound(sourceLen). Upon exit, destLen is the actual size of the + compressed buffer. + + compress returns Z_OK if success, Z_MEM_ERROR if there was not + enough memory, Z_BUF_ERROR if there was not enough room in the output + buffer. +*/ + +ZEXTERN int ZEXPORT compress2 OF((Bytef *dest, uLongf *destLen, + const Bytef *source, uLong sourceLen, + int level)); +/* + Compresses the source buffer into the destination buffer. The level + parameter has the same meaning as in deflateInit. sourceLen is the byte + length of the source buffer. Upon entry, destLen is the total size of the + destination buffer, which must be at least the value returned by + compressBound(sourceLen). Upon exit, destLen is the actual size of the + compressed buffer. + + compress2 returns Z_OK if success, Z_MEM_ERROR if there was not enough + memory, Z_BUF_ERROR if there was not enough room in the output buffer, + Z_STREAM_ERROR if the level parameter is invalid. +*/ + +ZEXTERN uLong ZEXPORT compressBound OF((uLong sourceLen)); +/* + compressBound() returns an upper bound on the compressed size after + compress() or compress2() on sourceLen bytes. It would be used before a + compress() or compress2() call to allocate the destination buffer. +*/ + +ZEXTERN int ZEXPORT uncompress OF((Bytef *dest, uLongf *destLen, + const Bytef *source, uLong sourceLen)); +/* + Decompresses the source buffer into the destination buffer. sourceLen is + the byte length of the source buffer. Upon entry, destLen is the total size + of the destination buffer, which must be large enough to hold the entire + uncompressed data. (The size of the uncompressed data must have been saved + previously by the compressor and transmitted to the decompressor by some + mechanism outside the scope of this compression library.) Upon exit, destLen + is the actual size of the uncompressed buffer. + + uncompress returns Z_OK if success, Z_MEM_ERROR if there was not + enough memory, Z_BUF_ERROR if there was not enough room in the output + buffer, or Z_DATA_ERROR if the input data was corrupted or incomplete. +*/ + + + /* gzip file access functions */ + +/* + This library supports reading and writing files in gzip (.gz) format with + an interface similar to that of stdio, using the functions that start with + "gz". The gzip format is different from the zlib format. gzip is a gzip + wrapper, documented in RFC 1952, wrapped around a deflate stream. +*/ + +typedef voidp gzFile; /* opaque gzip file descriptor */ + +/* +ZEXTERN gzFile ZEXPORT gzopen OF((const char *path, const char *mode)); + + Opens a gzip (.gz) file for reading or writing. The mode parameter is as + in fopen ("rb" or "wb") but can also include a compression level ("wb9") or + a strategy: 'f' for filtered data as in "wb6f", 'h' for Huffman-only + compression as in "wb1h", 'R' for run-length encoding as in "wb1R", or 'F' + for fixed code compression as in "wb9F". (See the description of + deflateInit2 for more information about the strategy parameter.) Also "a" + can be used instead of "w" to request that the gzip stream that will be + written be appended to the file. "+" will result in an error, since reading + and writing to the same gzip file is not supported. + + gzopen can be used to read a file which is not in gzip format; in this + case gzread will directly read from the file without decompression. + + gzopen returns NULL if the file could not be opened, if there was + insufficient memory to allocate the gzFile state, or if an invalid mode was + specified (an 'r', 'w', or 'a' was not provided, or '+' was provided). + errno can be checked to determine if the reason gzopen failed was that the + file could not be opened. +*/ + +ZEXTERN gzFile ZEXPORT gzdopen OF((int fd, const char *mode)); +/* + gzdopen associates a gzFile with the file descriptor fd. File descriptors + are obtained from calls like open, dup, creat, pipe or fileno (if the file + has been previously opened with fopen). The mode parameter is as in gzopen. + + The next call of gzclose on the returned gzFile will also close the file + descriptor fd, just like fclose(fdopen(fd, mode)) closes the file descriptor + fd. If you want to keep fd open, use fd = dup(fd_keep); gz = gzdopen(fd, + mode);. The duplicated descriptor should be saved to avoid a leak, since + gzdopen does not close fd if it fails. + + gzdopen returns NULL if there was insufficient memory to allocate the + gzFile state, if an invalid mode was specified (an 'r', 'w', or 'a' was not + provided, or '+' was provided), or if fd is -1. The file descriptor is not + used until the next gz* read, write, seek, or close operation, so gzdopen + will not detect if fd is invalid (unless fd is -1). +*/ + +ZEXTERN int ZEXPORT gzbuffer OF((gzFile file, unsigned size)); +/* + Set the internal buffer size used by this library's functions. The + default buffer size is 8192 bytes. This function must be called after + gzopen() or gzdopen(), and before any other calls that read or write the + file. The buffer memory allocation is always deferred to the first read or + write. Two buffers are allocated, either both of the specified size when + writing, or one of the specified size and the other twice that size when + reading. A larger buffer size of, for example, 64K or 128K bytes will + noticeably increase the speed of decompression (reading). + + The new buffer size also affects the maximum length for gzprintf(). + + gzbuffer() returns 0 on success, or -1 on failure, such as being called + too late. +*/ + +ZEXTERN int ZEXPORT gzsetparams OF((gzFile file, int level, int strategy)); +/* + Dynamically update the compression level or strategy. See the description + of deflateInit2 for the meaning of these parameters. + + gzsetparams returns Z_OK if success, or Z_STREAM_ERROR if the file was not + opened for writing. +*/ + +ZEXTERN int ZEXPORT gzread OF((gzFile file, voidp buf, unsigned len)); +/* + Reads the given number of uncompressed bytes from the compressed file. If + the input file was not in gzip format, gzread copies the given number of + bytes into the buffer. + + After reaching the end of a gzip stream in the input, gzread will continue + to read, looking for another gzip stream, or failing that, reading the rest + of the input file directly without decompression. The entire input file + will be read if gzread is called until it returns less than the requested + len. + + gzread returns the number of uncompressed bytes actually read, less than + len for end of file, or -1 for error. +*/ + +ZEXTERN int ZEXPORT gzwrite OF((gzFile file, + voidpc buf, unsigned len)); +/* + Writes the given number of uncompressed bytes into the compressed file. + gzwrite returns the number of uncompressed bytes written or 0 in case of + error. +*/ + +ZEXTERN int ZEXPORTVA gzprintf OF((gzFile file, const char *format, ...)); +/* + Converts, formats, and writes the arguments to the compressed file under + control of the format string, as in fprintf. gzprintf returns the number of + uncompressed bytes actually written, or 0 in case of error. The number of + uncompressed bytes written is limited to 8191, or one less than the buffer + size given to gzbuffer(). The caller should assure that this limit is not + exceeded. If it is exceeded, then gzprintf() will return an error (0) with + nothing written. In this case, there may also be a buffer overflow with + unpredictable consequences, which is possible only if zlib was compiled with + the insecure functions sprintf() or vsprintf() because the secure snprintf() + or vsnprintf() functions were not available. This can be determined using + zlibCompileFlags(). +*/ + +ZEXTERN int ZEXPORT gzputs OF((gzFile file, const char *s)); +/* + Writes the given null-terminated string to the compressed file, excluding + the terminating null character. + + gzputs returns the number of characters written, or -1 in case of error. +*/ + +ZEXTERN char * ZEXPORT gzgets OF((gzFile file, char *buf, int len)); +/* + Reads bytes from the compressed file until len-1 characters are read, or a + newline character is read and transferred to buf, or an end-of-file + condition is encountered. If any characters are read or if len == 1, the + string is terminated with a null character. If no characters are read due + to an end-of-file or len < 1, then the buffer is left untouched. + + gzgets returns buf which is a null-terminated string, or it returns NULL + for end-of-file or in case of error. If there was an error, the contents at + buf are indeterminate. +*/ + +ZEXTERN int ZEXPORT gzputc OF((gzFile file, int c)); +/* + Writes c, converted to an unsigned char, into the compressed file. gzputc + returns the value that was written, or -1 in case of error. +*/ + +ZEXTERN int ZEXPORT gzgetc OF((gzFile file)); +/* + Reads one byte from the compressed file. gzgetc returns this byte or -1 + in case of end of file or error. +*/ + +ZEXTERN int ZEXPORT gzungetc OF((int c, gzFile file)); +/* + Push one character back onto the stream to be read as the first character + on the next read. At least one character of push-back is allowed. + gzungetc() returns the character pushed, or -1 on failure. gzungetc() will + fail if c is -1, and may fail if a character has been pushed but not read + yet. If gzungetc is used immediately after gzopen or gzdopen, at least the + output buffer size of pushed characters is allowed. (See gzbuffer above.) + The pushed character will be discarded if the stream is repositioned with + gzseek() or gzrewind(). +*/ + +ZEXTERN int ZEXPORT gzflush OF((gzFile file, int flush)); +/* + Flushes all pending output into the compressed file. The parameter flush + is as in the deflate() function. The return value is the zlib error number + (see function gzerror below). gzflush is only permitted when writing. + + If the flush parameter is Z_FINISH, the remaining data is written and the + gzip stream is completed in the output. If gzwrite() is called again, a new + gzip stream will be started in the output. gzread() is able to read such + concatented gzip streams. + + gzflush should be called only when strictly necessary because it will + degrade compression if called too often. +*/ + +/* +ZEXTERN z_off_t ZEXPORT gzseek OF((gzFile file, + z_off_t offset, int whence)); + + Sets the starting position for the next gzread or gzwrite on the given + compressed file. The offset represents a number of bytes in the + uncompressed data stream. The whence parameter is defined as in lseek(2); + the value SEEK_END is not supported. + + If the file is opened for reading, this function is emulated but can be + extremely slow. If the file is opened for writing, only forward seeks are + supported; gzseek then compresses a sequence of zeroes up to the new + starting position. + + gzseek returns the resulting offset location as measured in bytes from + the beginning of the uncompressed stream, or -1 in case of error, in + particular if the file is opened for writing and the new starting position + would be before the current position. +*/ + +ZEXTERN int ZEXPORT gzrewind OF((gzFile file)); +/* + Rewinds the given file. This function is supported only for reading. + + gzrewind(file) is equivalent to (int)gzseek(file, 0L, SEEK_SET) +*/ + +/* +ZEXTERN z_off_t ZEXPORT gztell OF((gzFile file)); + + Returns the starting position for the next gzread or gzwrite on the given + compressed file. This position represents a number of bytes in the + uncompressed data stream, and is zero when starting, even if appending or + reading a gzip stream from the middle of a file using gzdopen(). + + gztell(file) is equivalent to gzseek(file, 0L, SEEK_CUR) +*/ + +/* +ZEXTERN z_off_t ZEXPORT gzoffset OF((gzFile file)); + + Returns the current offset in the file being read or written. This offset + includes the count of bytes that precede the gzip stream, for example when + appending or when using gzdopen() for reading. When reading, the offset + does not include as yet unused buffered input. This information can be used + for a progress indicator. On error, gzoffset() returns -1. +*/ + +ZEXTERN int ZEXPORT gzeof OF((gzFile file)); +/* + Returns true (1) if the end-of-file indicator has been set while reading, + false (0) otherwise. Note that the end-of-file indicator is set only if the + read tried to go past the end of the input, but came up short. Therefore, + just like feof(), gzeof() may return false even if there is no more data to + read, in the event that the last read request was for the exact number of + bytes remaining in the input file. This will happen if the input file size + is an exact multiple of the buffer size. + + If gzeof() returns true, then the read functions will return no more data, + unless the end-of-file indicator is reset by gzclearerr() and the input file + has grown since the previous end of file was detected. +*/ + +ZEXTERN int ZEXPORT gzdirect OF((gzFile file)); +/* + Returns true (1) if file is being copied directly while reading, or false + (0) if file is a gzip stream being decompressed. This state can change from + false to true while reading the input file if the end of a gzip stream is + reached, but is followed by data that is not another gzip stream. + + If the input file is empty, gzdirect() will return true, since the input + does not contain a gzip stream. + + If gzdirect() is used immediately after gzopen() or gzdopen() it will + cause buffers to be allocated to allow reading the file to determine if it + is a gzip file. Therefore if gzbuffer() is used, it should be called before + gzdirect(). +*/ + +ZEXTERN int ZEXPORT gzclose OF((gzFile file)); +/* + Flushes all pending output if necessary, closes the compressed file and + deallocates the (de)compression state. Note that once file is closed, you + cannot call gzerror with file, since its structures have been deallocated. + gzclose must not be called more than once on the same file, just as free + must not be called more than once on the same allocation. + + gzclose will return Z_STREAM_ERROR if file is not valid, Z_ERRNO on a + file operation error, or Z_OK on success. +*/ + +ZEXTERN int ZEXPORT gzclose_r OF((gzFile file)); +ZEXTERN int ZEXPORT gzclose_w OF((gzFile file)); +/* + Same as gzclose(), but gzclose_r() is only for use when reading, and + gzclose_w() is only for use when writing or appending. The advantage to + using these instead of gzclose() is that they avoid linking in zlib + compression or decompression code that is not used when only reading or only + writing respectively. If gzclose() is used, then both compression and + decompression code will be included the application when linking to a static + zlib library. +*/ + +ZEXTERN const char * ZEXPORT gzerror OF((gzFile file, int *errnum)); +/* + Returns the error message for the last error which occurred on the given + compressed file. errnum is set to zlib error number. If an error occurred + in the file system and not in the compression library, errnum is set to + Z_ERRNO and the application may consult errno to get the exact error code. + + The application must not modify the returned string. Future calls to + this function may invalidate the previously returned string. If file is + closed, then the string previously returned by gzerror will no longer be + available. + + gzerror() should be used to distinguish errors from end-of-file for those + functions above that do not distinguish those cases in their return values. +*/ + +ZEXTERN void ZEXPORT gzclearerr OF((gzFile file)); +/* + Clears the error and end-of-file flags for file. This is analogous to the + clearerr() function in stdio. This is useful for continuing to read a gzip + file that is being written concurrently. +*/ + + + /* checksum functions */ + +/* + These functions are not related to compression but are exported + anyway because they might be useful in applications using the compression + library. +*/ + +ZEXTERN uLong ZEXPORT adler32 OF((uLong adler, const Bytef *buf, uInt len)); +/* + Update a running Adler-32 checksum with the bytes buf[0..len-1] and + return the updated checksum. If buf is Z_NULL, this function returns the + required initial value for the checksum. + + An Adler-32 checksum is almost as reliable as a CRC32 but can be computed + much faster. + + Usage example: + + uLong adler = adler32(0L, Z_NULL, 0); + + while (read_buffer(buffer, length) != EOF) { + adler = adler32(adler, buffer, length); + } + if (adler != original_adler) error(); +*/ + +/* +ZEXTERN uLong ZEXPORT adler32_combine OF((uLong adler1, uLong adler2, + z_off_t len2)); + + Combine two Adler-32 checksums into one. For two sequences of bytes, seq1 + and seq2 with lengths len1 and len2, Adler-32 checksums were calculated for + each, adler1 and adler2. adler32_combine() returns the Adler-32 checksum of + seq1 and seq2 concatenated, requiring only adler1, adler2, and len2. +*/ + +ZEXTERN uLong ZEXPORT crc32 OF((uLong crc, const Bytef *buf, uInt len)); +/* + Update a running CRC-32 with the bytes buf[0..len-1] and return the + updated CRC-32. If buf is Z_NULL, this function returns the required + initial value for the for the crc. Pre- and post-conditioning (one's + complement) is performed within this function so it shouldn't be done by the + application. + + Usage example: + + uLong crc = crc32(0L, Z_NULL, 0); + + while (read_buffer(buffer, length) != EOF) { + crc = crc32(crc, buffer, length); + } + if (crc != original_crc) error(); +*/ + +/* +ZEXTERN uLong ZEXPORT crc32_combine OF((uLong crc1, uLong crc2, z_off_t len2)); + + Combine two CRC-32 check values into one. For two sequences of bytes, + seq1 and seq2 with lengths len1 and len2, CRC-32 check values were + calculated for each, crc1 and crc2. crc32_combine() returns the CRC-32 + check value of seq1 and seq2 concatenated, requiring only crc1, crc2, and + len2. +*/ + + + /* various hacks, don't look :) */ + +/* deflateInit and inflateInit are macros to allow checking the zlib version + * and the compiler's view of z_stream: + */ +ZEXTERN int ZEXPORT deflateInit_ OF((z_streamp strm, int level, + const char *version, int stream_size)); +ZEXTERN int ZEXPORT inflateInit_ OF((z_streamp strm, + const char *version, int stream_size)); +ZEXTERN int ZEXPORT deflateInit2_ OF((z_streamp strm, int level, int method, + int windowBits, int memLevel, + int strategy, const char *version, + int stream_size)); +ZEXTERN int ZEXPORT inflateInit2_ OF((z_streamp strm, int windowBits, + const char *version, int stream_size)); +ZEXTERN int ZEXPORT inflateBackInit_ OF((z_streamp strm, int windowBits, + unsigned char FAR *window, + const char *version, + int stream_size)); +#define deflateInit(strm, level) \ + deflateInit_((strm), (level), ZLIB_VERSION, sizeof(z_stream)) +#define inflateInit(strm) \ + inflateInit_((strm), ZLIB_VERSION, sizeof(z_stream)) +#define deflateInit2(strm, level, method, windowBits, memLevel, strategy) \ + deflateInit2_((strm),(level),(method),(windowBits),(memLevel),\ + (strategy), ZLIB_VERSION, sizeof(z_stream)) +#define inflateInit2(strm, windowBits) \ + inflateInit2_((strm), (windowBits), ZLIB_VERSION, sizeof(z_stream)) +#define inflateBackInit(strm, windowBits, window) \ + inflateBackInit_((strm), (windowBits), (window), \ + ZLIB_VERSION, sizeof(z_stream)) + +/* provide 64-bit offset functions if _LARGEFILE64_SOURCE defined, and/or + * change the regular functions to 64 bits if _FILE_OFFSET_BITS is 64 (if + * both are true, the application gets the *64 functions, and the regular + * functions are changed to 64 bits) -- in case these are set on systems + * without large file support, _LFS64_LARGEFILE must also be true + */ +#if defined(_LARGEFILE64_SOURCE) && _LFS64_LARGEFILE-0 + ZEXTERN gzFile ZEXPORT gzopen64 OF((const char *, const char *)); + ZEXTERN z_off64_t ZEXPORT gzseek64 OF((gzFile, z_off64_t, int)); + ZEXTERN z_off64_t ZEXPORT gztell64 OF((gzFile)); + ZEXTERN z_off64_t ZEXPORT gzoffset64 OF((gzFile)); + ZEXTERN uLong ZEXPORT adler32_combine64 OF((uLong, uLong, z_off64_t)); + ZEXTERN uLong ZEXPORT crc32_combine64 OF((uLong, uLong, z_off64_t)); +#endif + +#if !defined(ZLIB_INTERNAL) && _FILE_OFFSET_BITS-0 == 64 && _LFS64_LARGEFILE-0 +# define gzopen gzopen64 +# define gzseek gzseek64 +# define gztell gztell64 +# define gzoffset gzoffset64 +# define adler32_combine adler32_combine64 +# define crc32_combine crc32_combine64 +# ifdef _LARGEFILE64_SOURCE + ZEXTERN gzFile ZEXPORT gzopen64 OF((const char *, const char *)); + ZEXTERN z_off_t ZEXPORT gzseek64 OF((gzFile, z_off_t, int)); + ZEXTERN z_off_t ZEXPORT gztell64 OF((gzFile)); + ZEXTERN z_off_t ZEXPORT gzoffset64 OF((gzFile)); + ZEXTERN uLong ZEXPORT adler32_combine64 OF((uLong, uLong, z_off_t)); + ZEXTERN uLong ZEXPORT crc32_combine64 OF((uLong, uLong, z_off_t)); +# endif +#else + ZEXTERN gzFile ZEXPORT gzopen OF((const char *, const char *)); + ZEXTERN z_off_t ZEXPORT gzseek OF((gzFile, z_off_t, int)); + ZEXTERN z_off_t ZEXPORT gztell OF((gzFile)); + ZEXTERN z_off_t ZEXPORT gzoffset OF((gzFile)); + ZEXTERN uLong ZEXPORT adler32_combine OF((uLong, uLong, z_off_t)); + ZEXTERN uLong ZEXPORT crc32_combine OF((uLong, uLong, z_off_t)); +#endif + +/* hack for buggy compilers */ +#if !defined(ZUTIL_H) && !defined(NO_DUMMY_DECL) + struct internal_state {int dummy;}; +#endif + +/* undocumented functions */ +ZEXTERN const char * ZEXPORT zError OF((int)); +ZEXTERN int ZEXPORT inflateSyncPoint OF((z_streamp)); +ZEXTERN const uLongf * ZEXPORT get_crc_table OF((void)); +ZEXTERN int ZEXPORT inflateUndermine OF((z_streamp, int)); + +#ifdef __cplusplus +} +#endif + +#endif /* ZLIB_H */ diff --git a/deps/zlib/zutil.c b/deps/zlib/zutil.c new file mode 100644 index 000000000..898ed345b --- /dev/null +++ b/deps/zlib/zutil.c @@ -0,0 +1,318 @@ +/* zutil.c -- target dependent utility functions for the compression library + * Copyright (C) 1995-2005, 2010 Jean-loup Gailly. + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* @(#) $Id$ */ + +#include "zutil.h" + +#ifndef NO_DUMMY_DECL +struct internal_state {int dummy;}; /* for buggy compilers */ +#endif + +const char * const z_errmsg[10] = { +"need dictionary", /* Z_NEED_DICT 2 */ +"stream end", /* Z_STREAM_END 1 */ +"", /* Z_OK 0 */ +"file error", /* Z_ERRNO (-1) */ +"stream error", /* Z_STREAM_ERROR (-2) */ +"data error", /* Z_DATA_ERROR (-3) */ +"insufficient memory", /* Z_MEM_ERROR (-4) */ +"buffer error", /* Z_BUF_ERROR (-5) */ +"incompatible version",/* Z_VERSION_ERROR (-6) */ +""}; + + +const char * ZEXPORT zlibVersion() +{ + return ZLIB_VERSION; +} + +uLong ZEXPORT zlibCompileFlags() +{ + uLong flags; + + flags = 0; + switch ((int)(sizeof(uInt))) { + case 2: break; + case 4: flags += 1; break; + case 8: flags += 2; break; + default: flags += 3; + } + switch ((int)(sizeof(uLong))) { + case 2: break; + case 4: flags += 1 << 2; break; + case 8: flags += 2 << 2; break; + default: flags += 3 << 2; + } + switch ((int)(sizeof(voidpf))) { + case 2: break; + case 4: flags += 1 << 4; break; + case 8: flags += 2 << 4; break; + default: flags += 3 << 4; + } + switch ((int)(sizeof(z_off_t))) { + case 2: break; + case 4: flags += 1 << 6; break; + case 8: flags += 2 << 6; break; + default: flags += 3 << 6; + } +#ifdef DEBUG + flags += 1 << 8; +#endif +#if defined(ASMV) || defined(ASMINF) + flags += 1 << 9; +#endif +#ifdef ZLIB_WINAPI + flags += 1 << 10; +#endif +#ifdef BUILDFIXED + flags += 1 << 12; +#endif +#ifdef DYNAMIC_CRC_TABLE + flags += 1 << 13; +#endif +#ifdef NO_GZCOMPRESS + flags += 1L << 16; +#endif +#ifdef NO_GZIP + flags += 1L << 17; +#endif +#ifdef PKZIP_BUG_WORKAROUND + flags += 1L << 20; +#endif +#ifdef FASTEST + flags += 1L << 21; +#endif +#ifdef STDC +# ifdef NO_vsnprintf + flags += 1L << 25; +# ifdef HAS_vsprintf_void + flags += 1L << 26; +# endif +# else +# ifdef HAS_vsnprintf_void + flags += 1L << 26; +# endif +# endif +#else + flags += 1L << 24; +# ifdef NO_snprintf + flags += 1L << 25; +# ifdef HAS_sprintf_void + flags += 1L << 26; +# endif +# else +# ifdef HAS_snprintf_void + flags += 1L << 26; +# endif +# endif +#endif + return flags; +} + +#ifdef DEBUG + +# ifndef verbose +# define verbose 0 +# endif +int ZLIB_INTERNAL z_verbose = verbose; + +void ZLIB_INTERNAL z_error (m) + char *m; +{ + fprintf(stderr, "%s\n", m); + exit(1); +} +#endif + +/* exported to allow conversion of error code to string for compress() and + * uncompress() + */ +const char * ZEXPORT zError(err) + int err; +{ + return ERR_MSG(err); +} + +#if defined(_WIN32_WCE) + /* The Microsoft C Run-Time Library for Windows CE doesn't have + * errno. We define it as a global variable to simplify porting. + * Its value is always 0 and should not be used. + */ + int errno = 0; +#endif + +#ifndef HAVE_MEMCPY + +void ZLIB_INTERNAL zmemcpy(dest, source, len) + Bytef* dest; + const Bytef* source; + uInt len; +{ + if (len == 0) return; + do { + *dest++ = *source++; /* ??? to be unrolled */ + } while (--len != 0); +} + +int ZLIB_INTERNAL zmemcmp(s1, s2, len) + const Bytef* s1; + const Bytef* s2; + uInt len; +{ + uInt j; + + for (j = 0; j < len; j++) { + if (s1[j] != s2[j]) return 2*(s1[j] > s2[j])-1; + } + return 0; +} + +void ZLIB_INTERNAL zmemzero(dest, len) + Bytef* dest; + uInt len; +{ + if (len == 0) return; + do { + *dest++ = 0; /* ??? to be unrolled */ + } while (--len != 0); +} +#endif + + +#ifdef SYS16BIT + +#ifdef __TURBOC__ +/* Turbo C in 16-bit mode */ + +# define MY_ZCALLOC + +/* Turbo C malloc() does not allow dynamic allocation of 64K bytes + * and farmalloc(64K) returns a pointer with an offset of 8, so we + * must fix the pointer. Warning: the pointer must be put back to its + * original form in order to free it, use zcfree(). + */ + +#define MAX_PTR 10 +/* 10*64K = 640K */ + +local int next_ptr = 0; + +typedef struct ptr_table_s { + voidpf org_ptr; + voidpf new_ptr; +} ptr_table; + +local ptr_table table[MAX_PTR]; +/* This table is used to remember the original form of pointers + * to large buffers (64K). Such pointers are normalized with a zero offset. + * Since MSDOS is not a preemptive multitasking OS, this table is not + * protected from concurrent access. This hack doesn't work anyway on + * a protected system like OS/2. Use Microsoft C instead. + */ + +voidpf ZLIB_INTERNAL zcalloc (voidpf opaque, unsigned items, unsigned size) +{ + voidpf buf = opaque; /* just to make some compilers happy */ + ulg bsize = (ulg)items*size; + + /* If we allocate less than 65520 bytes, we assume that farmalloc + * will return a usable pointer which doesn't have to be normalized. + */ + if (bsize < 65520L) { + buf = farmalloc(bsize); + if (*(ush*)&buf != 0) return buf; + } else { + buf = farmalloc(bsize + 16L); + } + if (buf == NULL || next_ptr >= MAX_PTR) return NULL; + table[next_ptr].org_ptr = buf; + + /* Normalize the pointer to seg:0 */ + *((ush*)&buf+1) += ((ush)((uch*)buf-0) + 15) >> 4; + *(ush*)&buf = 0; + table[next_ptr++].new_ptr = buf; + return buf; +} + +void ZLIB_INTERNAL zcfree (voidpf opaque, voidpf ptr) +{ + int n; + if (*(ush*)&ptr != 0) { /* object < 64K */ + farfree(ptr); + return; + } + /* Find the original pointer */ + for (n = 0; n < next_ptr; n++) { + if (ptr != table[n].new_ptr) continue; + + farfree(table[n].org_ptr); + while (++n < next_ptr) { + table[n-1] = table[n]; + } + next_ptr--; + return; + } + ptr = opaque; /* just to make some compilers happy */ + Assert(0, "zcfree: ptr not found"); +} + +#endif /* __TURBOC__ */ + + +#ifdef M_I86 +/* Microsoft C in 16-bit mode */ + +# define MY_ZCALLOC + +#if (!defined(_MSC_VER) || (_MSC_VER <= 600)) +# define _halloc halloc +# define _hfree hfree +#endif + +voidpf ZLIB_INTERNAL zcalloc (voidpf opaque, uInt items, uInt size) +{ + if (opaque) opaque = 0; /* to make compiler happy */ + return _halloc((long)items, size); +} + +void ZLIB_INTERNAL zcfree (voidpf opaque, voidpf ptr) +{ + if (opaque) opaque = 0; /* to make compiler happy */ + _hfree(ptr); +} + +#endif /* M_I86 */ + +#endif /* SYS16BIT */ + + +#ifndef MY_ZCALLOC /* Any system without a special alloc function */ + +#ifndef STDC +extern voidp malloc OF((uInt size)); +extern voidp calloc OF((uInt items, uInt size)); +extern void free OF((voidpf ptr)); +#endif + +voidpf ZLIB_INTERNAL zcalloc (opaque, items, size) + voidpf opaque; + unsigned items; + unsigned size; +{ + if (opaque) items += size - size; /* make compiler happy */ + return sizeof(uInt) > 2 ? (voidpf)malloc(items * size) : + (voidpf)calloc(items, size); +} + +void ZLIB_INTERNAL zcfree (opaque, ptr) + voidpf opaque; + voidpf ptr; +{ + free(ptr); + if (opaque) return; /* make compiler happy */ +} + +#endif /* MY_ZCALLOC */ diff --git a/deps/zlib/zutil.h b/deps/zlib/zutil.h new file mode 100644 index 000000000..258fa8879 --- /dev/null +++ b/deps/zlib/zutil.h @@ -0,0 +1,274 @@ +/* zutil.h -- internal interface and configuration of the compression library + * Copyright (C) 1995-2010 Jean-loup Gailly. + * For conditions of distribution and use, see copyright notice in zlib.h + */ + +/* WARNING: this file should *not* be used by applications. It is + part of the implementation of the compression library and is + subject to change. Applications should only use zlib.h. + */ + +/* @(#) $Id$ */ + +#ifndef ZUTIL_H +#define ZUTIL_H + +#if ((__GNUC__-0) * 10 + __GNUC_MINOR__-0 >= 33) && !defined(NO_VIZ) +# define ZLIB_INTERNAL __attribute__((visibility ("hidden"))) +#else +# define ZLIB_INTERNAL +#endif + +#include "zlib.h" + +#ifdef STDC +# if !(defined(_WIN32_WCE) && defined(_MSC_VER)) +# include +# endif +# include +# include +#endif + +#ifndef local +# define local static +#endif +/* compile with -Dlocal if your debugger can't find static symbols */ + +typedef unsigned char uch; +typedef uch FAR uchf; +typedef unsigned short ush; +typedef ush FAR ushf; +typedef unsigned long ulg; + +extern const char * const z_errmsg[10]; /* indexed by 2-zlib_error */ +/* (size given to avoid silly warnings with Visual C++) */ + +#define ERR_MSG(err) z_errmsg[Z_NEED_DICT-(err)] + +#define ERR_RETURN(strm,err) \ + return (strm->msg = (char*)ERR_MSG(err), (err)) +/* To be used only when the state is known to be valid */ + + /* common constants */ + +#ifndef DEF_WBITS +# define DEF_WBITS MAX_WBITS +#endif +/* default windowBits for decompression. MAX_WBITS is for compression only */ + +#if MAX_MEM_LEVEL >= 8 +# define DEF_MEM_LEVEL 8 +#else +# define DEF_MEM_LEVEL MAX_MEM_LEVEL +#endif +/* default memLevel */ + +#define STORED_BLOCK 0 +#define STATIC_TREES 1 +#define DYN_TREES 2 +/* The three kinds of block type */ + +#define MIN_MATCH 3 +#define MAX_MATCH 258 +/* The minimum and maximum match lengths */ + +#define PRESET_DICT 0x20 /* preset dictionary flag in zlib header */ + + /* target dependencies */ + +#if defined(MSDOS) || (defined(WINDOWS) && !defined(WIN32)) +# define OS_CODE 0x00 +# if defined(__TURBOC__) || defined(__BORLANDC__) +# if (__STDC__ == 1) && (defined(__LARGE__) || defined(__COMPACT__)) + /* Allow compilation with ANSI keywords only enabled */ + void _Cdecl farfree( void *block ); + void *_Cdecl farmalloc( unsigned long nbytes ); +# else +# include +# endif +# else /* MSC or DJGPP */ +# include +# endif +#endif + +#ifdef AMIGA +# define OS_CODE 0x01 +#endif + +#if defined(VAXC) || defined(VMS) +# define OS_CODE 0x02 +# define F_OPEN(name, mode) \ + fopen((name), (mode), "mbc=60", "ctx=stm", "rfm=fix", "mrs=512") +#endif + +#if defined(ATARI) || defined(atarist) +# define OS_CODE 0x05 +#endif + +#ifdef OS2 +# define OS_CODE 0x06 +# ifdef M_I86 +# include +# endif +#endif + +#if defined(MACOS) || defined(TARGET_OS_MAC) +# define OS_CODE 0x07 +# if defined(__MWERKS__) && __dest_os != __be_os && __dest_os != __win32_os +# include /* for fdopen */ +# else +# ifndef fdopen +# define fdopen(fd,mode) NULL /* No fdopen() */ +# endif +# endif +#endif + +#ifdef TOPS20 +# define OS_CODE 0x0a +#endif + +#ifdef WIN32 +# ifndef __CYGWIN__ /* Cygwin is Unix, not Win32 */ +# define OS_CODE 0x0b +# endif +#endif + +#ifdef __50SERIES /* Prime/PRIMOS */ +# define OS_CODE 0x0f +#endif + +#if defined(_BEOS_) || defined(RISCOS) +# define fdopen(fd,mode) NULL /* No fdopen() */ +#endif + +#if (defined(_MSC_VER) && (_MSC_VER > 600)) && !defined __INTERIX +# if defined(_WIN32_WCE) +# define fdopen(fd,mode) NULL /* No fdopen() */ +# ifndef _PTRDIFF_T_DEFINED + typedef int ptrdiff_t; +# define _PTRDIFF_T_DEFINED +# endif +# else +# define fdopen(fd,type) _fdopen(fd,type) +# endif +#endif + +#if defined(__BORLANDC__) + #pragma warn -8004 + #pragma warn -8008 + #pragma warn -8066 +#endif + +/* provide prototypes for these when building zlib without LFS */ +#if !defined(_LARGEFILE64_SOURCE) || _LFS64_LARGEFILE-0 == 0 + ZEXTERN uLong ZEXPORT adler32_combine64 OF((uLong, uLong, z_off_t)); + ZEXTERN uLong ZEXPORT crc32_combine64 OF((uLong, uLong, z_off_t)); +#endif + + /* common defaults */ + +#ifndef OS_CODE +# define OS_CODE 0x03 /* assume Unix */ +#endif + +#ifndef F_OPEN +# define F_OPEN(name, mode) fopen((name), (mode)) +#endif + + /* functions */ + +#if defined(STDC99) || (defined(__TURBOC__) && __TURBOC__ >= 0x550) +# ifndef HAVE_VSNPRINTF +# define HAVE_VSNPRINTF +# endif +#endif +#if defined(__CYGWIN__) +# ifndef HAVE_VSNPRINTF +# define HAVE_VSNPRINTF +# endif +#endif +#ifndef HAVE_VSNPRINTF +# ifdef MSDOS + /* vsnprintf may exist on some MS-DOS compilers (DJGPP?), + but for now we just assume it doesn't. */ +# define NO_vsnprintf +# endif +# ifdef __TURBOC__ +# define NO_vsnprintf +# endif +# ifdef WIN32 + /* In Win32, vsnprintf is available as the "non-ANSI" _vsnprintf. */ +# if !defined(vsnprintf) && !defined(NO_vsnprintf) +# if !defined(_MSC_VER) || ( defined(_MSC_VER) && _MSC_VER < 1500 ) +# define vsnprintf _vsnprintf +# endif +# endif +# endif +# ifdef __SASC +# define NO_vsnprintf +# endif +#endif +#ifdef VMS +# define NO_vsnprintf +#endif + +#if defined(pyr) +# define NO_MEMCPY +#endif +#if defined(SMALL_MEDIUM) && !defined(_MSC_VER) && !defined(__SC__) + /* Use our own functions for small and medium model with MSC <= 5.0. + * You may have to use the same strategy for Borland C (untested). + * The __SC__ check is for Symantec. + */ +# define NO_MEMCPY +#endif +#if defined(STDC) && !defined(HAVE_MEMCPY) && !defined(NO_MEMCPY) +# define HAVE_MEMCPY +#endif +#ifdef HAVE_MEMCPY +# ifdef SMALL_MEDIUM /* MSDOS small or medium model */ +# define zmemcpy _fmemcpy +# define zmemcmp _fmemcmp +# define zmemzero(dest, len) _fmemset(dest, 0, len) +# else +# define zmemcpy memcpy +# define zmemcmp memcmp +# define zmemzero(dest, len) memset(dest, 0, len) +# endif +#else + void ZLIB_INTERNAL zmemcpy OF((Bytef* dest, const Bytef* source, uInt len)); + int ZLIB_INTERNAL zmemcmp OF((const Bytef* s1, const Bytef* s2, uInt len)); + void ZLIB_INTERNAL zmemzero OF((Bytef* dest, uInt len)); +#endif + +/* Diagnostic functions */ +#ifdef DEBUG +# include + extern int ZLIB_INTERNAL z_verbose; + extern void ZLIB_INTERNAL z_error OF((char *m)); +# define Assert(cond,msg) {if(!(cond)) z_error(msg);} +# define Trace(x) {if (z_verbose>=0) fprintf x ;} +# define Tracev(x) {if (z_verbose>0) fprintf x ;} +# define Tracevv(x) {if (z_verbose>1) fprintf x ;} +# define Tracec(c,x) {if (z_verbose>0 && (c)) fprintf x ;} +# define Tracecv(c,x) {if (z_verbose>1 && (c)) fprintf x ;} +#else +# define Assert(cond,msg) +# define Trace(x) +# define Tracev(x) +# define Tracevv(x) +# define Tracec(c,x) +# define Tracecv(c,x) +#endif + + +voidpf ZLIB_INTERNAL zcalloc OF((voidpf opaque, unsigned items, + unsigned size)); +void ZLIB_INTERNAL zcfree OF((voidpf opaque, voidpf ptr)); + +#define ZALLOC(strm, items, size) \ + (*((strm)->zalloc))((strm)->opaque, (items), (size)) +#define ZFREE(strm, addr) (*((strm)->zfree))((strm)->opaque, (voidpf)(addr)) +#define TRY_FREE(s, p) {if (p) ZFREE(s, p);} + +#endif /* ZUTIL_H */ diff --git a/wscript b/wscript index f9daca375..d791d932d 100644 --- a/wscript +++ b/wscript @@ -16,7 +16,7 @@ CFLAGS_WIN32_L = ['/RELEASE'] # used for /both/ debug and release builds. # sets the module's checksum in the header. CFLAGS_WIN32_L_DBG = ['/DEBUG'] -ALL_LIBS = ['z', 'crypto', 'pthread', 'sqlite3'] +ALL_LIBS = ['crypto', 'pthread', 'sqlite3'] def options(opt): opt.load('compiler_c') @@ -44,7 +44,6 @@ def configure(conf): conf.load('compiler_c') dbg = conf.options.debug - zlib_name = 'z' conf.env.CFLAGS = CFLAGS_UNIX + (CFLAGS_UNIX_DBG if dbg else []) @@ -56,8 +55,7 @@ def configure(conf): (CFLAGS_WIN32_DBG if dbg else CFLAGS_WIN32_RELEASE) conf.env.LINKFLAGS += CFLAGS_WIN32_L + \ (CFLAGS_WIN32_L_DBG if dbg else []) - conf.env.DEFINES += ['WIN32', '_DEBUG', '_LIB', 'ZLIB_WINAPI'] - zlib_name = 'zlibwapi' + conf.env.DEFINES += ['WIN32', '_DEBUG', '_LIB'] elif conf.env.CC_NAME == 'gcc': conf.check_cc(lib='pthread', uselib_store='pthread') @@ -65,8 +63,8 @@ def configure(conf): else: conf.env.PLATFORM = 'unix' - # check for Z lib - conf.check_cc(lib=zlib_name, uselib_store='z', install_path=None) + # Do not build ZLib with GZIP support + conf.env.DEFINES += ['NO_GZIP'] # check for sqlite3 if conf.options.use_sqlite and conf.check_cc( @@ -139,6 +137,7 @@ def build_library(bld, build_type): # src/win32/*.c sources = sources + directory.ant_glob('src/%s/*.c' % bld.env.PLATFORM) sources = sources + directory.ant_glob('src/backends/*.c') + sources = sources + directory.ant_glob('deps/zlib/*.c') # SHA1 methods source if bld.env.sha1 == "ppc": @@ -153,7 +152,7 @@ def build_library(bld, build_type): BUILD[build_type]( source=sources, target='git2', - includes=['src', 'include'], + includes=['src', 'include', 'deps/zlib'], install_path='${LIBDIR}', use=ALL_LIBS, vnum=version, From 11a203e1b174fc3ec923be9ea53d0451f75e21f5 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 15 Mar 2011 03:31:52 +0200 Subject: [PATCH 13/81] Add nodegit to the README --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index ee6d87bbb..c9eb6dc25 100644 --- a/README.md +++ b/README.md @@ -124,6 +124,7 @@ Here are the bindings to libgit2 that are currently available: * luagit2 (Lua bindings) * GitForDelphi (Delphi bindings) * node-gitteh (Node.js bindings) +* nodegit (Node.js bindings) * libqgit2 (C++ QT bindings) * Geef (Erlang bindings) From 5f8078d4dc325356e732c1a5e62acb7de04d72be Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 15 Mar 2011 04:03:54 +0200 Subject: [PATCH 14/81] Use a more sane zconf.f when building Zlib --- deps/zlib/zconf.h | 410 +++------------------------------------------- 1 file changed, 19 insertions(+), 391 deletions(-) diff --git a/deps/zlib/zconf.h b/deps/zlib/zconf.h index b23438744..ea765a54c 100644 --- a/deps/zlib/zconf.h +++ b/deps/zlib/zconf.h @@ -8,421 +8,49 @@ #ifndef ZCONF_H #define ZCONF_H -/* - * If you *really* need a unique prefix for all types and library functions, - * compile with -DZ_PREFIX. The "standard" zlib should be compiled without it. - * Even better than compiling with -DZ_PREFIX would be to use configure to set - * this permanently in zconf.h using "./configure --zprefix". - */ -#ifdef Z_PREFIX /* may be set to #if 1 by ./configure */ +#include "../../src/common.h" -/* all linked symbols */ -# define _dist_code z__dist_code -# define _length_code z__length_code -# define _tr_align z__tr_align -# define _tr_flush_block z__tr_flush_block -# define _tr_init z__tr_init -# define _tr_stored_block z__tr_stored_block -# define _tr_tally z__tr_tally -# define adler32 z_adler32 -# define adler32_combine z_adler32_combine -# define adler32_combine64 z_adler32_combine64 -# define compress z_compress -# define compress2 z_compress2 -# define compressBound z_compressBound -# define crc32 z_crc32 -# define crc32_combine z_crc32_combine -# define crc32_combine64 z_crc32_combine64 -# define deflate z_deflate -# define deflateBound z_deflateBound -# define deflateCopy z_deflateCopy -# define deflateEnd z_deflateEnd -# define deflateInit2_ z_deflateInit2_ -# define deflateInit_ z_deflateInit_ -# define deflateParams z_deflateParams -# define deflatePrime z_deflatePrime -# define deflateReset z_deflateReset -# define deflateSetDictionary z_deflateSetDictionary -# define deflateSetHeader z_deflateSetHeader -# define deflateTune z_deflateTune -# define deflate_copyright z_deflate_copyright -# define get_crc_table z_get_crc_table -# define gz_error z_gz_error -# define gz_intmax z_gz_intmax -# define gz_strwinerror z_gz_strwinerror -# define gzbuffer z_gzbuffer -# define gzclearerr z_gzclearerr -# define gzclose z_gzclose -# define gzclose_r z_gzclose_r -# define gzclose_w z_gzclose_w -# define gzdirect z_gzdirect -# define gzdopen z_gzdopen -# define gzeof z_gzeof -# define gzerror z_gzerror -# define gzflush z_gzflush -# define gzgetc z_gzgetc -# define gzgets z_gzgets -# define gzoffset z_gzoffset -# define gzoffset64 z_gzoffset64 -# define gzopen z_gzopen -# define gzopen64 z_gzopen64 -# define gzprintf z_gzprintf -# define gzputc z_gzputc -# define gzputs z_gzputs -# define gzread z_gzread -# define gzrewind z_gzrewind -# define gzseek z_gzseek -# define gzseek64 z_gzseek64 -# define gzsetparams z_gzsetparams -# define gztell z_gztell -# define gztell64 z_gztell64 -# define gzungetc z_gzungetc -# define gzwrite z_gzwrite -# define inflate z_inflate -# define inflateBack z_inflateBack -# define inflateBackEnd z_inflateBackEnd -# define inflateBackInit_ z_inflateBackInit_ -# define inflateCopy z_inflateCopy -# define inflateEnd z_inflateEnd -# define inflateGetHeader z_inflateGetHeader -# define inflateInit2_ z_inflateInit2_ -# define inflateInit_ z_inflateInit_ -# define inflateMark z_inflateMark -# define inflatePrime z_inflatePrime -# define inflateReset z_inflateReset -# define inflateReset2 z_inflateReset2 -# define inflateSetDictionary z_inflateSetDictionary -# define inflateSync z_inflateSync -# define inflateSyncPoint z_inflateSyncPoint -# define inflateUndermine z_inflateUndermine -# define inflate_copyright z_inflate_copyright -# define inflate_fast z_inflate_fast -# define inflate_table z_inflate_table -# define uncompress z_uncompress -# define zError z_zError -# define zcalloc z_zcalloc -# define zcfree z_zcfree -# define zlibCompileFlags z_zlibCompileFlags -# define zlibVersion z_zlibVersion +#define STDC -/* all zlib typedefs in zlib.h and zconf.h */ -# define Byte z_Byte -# define Bytef z_Bytef -# define alloc_func z_alloc_func -# define charf z_charf -# define free_func z_free_func -# define gzFile z_gzFile -# define gz_header z_gz_header -# define gz_headerp z_gz_headerp -# define in_func z_in_func -# define intf z_intf -# define out_func z_out_func -# define uInt z_uInt -# define uIntf z_uIntf -# define uLong z_uLong -# define uLongf z_uLongf -# define voidp z_voidp -# define voidpc z_voidpc -# define voidpf z_voidpf - -/* all zlib structs in zlib.h and zconf.h */ -# define gz_header_s z_gz_header_s -# define internal_state z_internal_state - -#endif - -#if defined(__MSDOS__) && !defined(MSDOS) -# define MSDOS -#endif -#if (defined(OS_2) || defined(__OS2__)) && !defined(OS2) -# define OS2 -#endif -#if defined(_WINDOWS) && !defined(WINDOWS) -# define WINDOWS -#endif -#if defined(_WIN32) || defined(_WIN32_WCE) || defined(__WIN32__) -# ifndef WIN32 -# define WIN32 -# endif -#endif -#if (defined(MSDOS) || defined(OS2) || defined(WINDOWS)) && !defined(WIN32) -# if !defined(__GNUC__) && !defined(__FLAT__) && !defined(__386__) -# ifndef SYS16BIT -# define SYS16BIT -# endif -# endif -#endif - -/* - * Compile with -DMAXSEG_64K if the alloc function cannot allocate more - * than 64k bytes at a time (needed on systems with 16-bit int). - */ -#ifdef SYS16BIT -# define MAXSEG_64K -#endif -#ifdef MSDOS -# define UNALIGNED_OK -#endif - -#ifdef __STDC_VERSION__ -# ifndef STDC -# define STDC -# endif -# if __STDC_VERSION__ >= 199901L -# ifndef STDC99 -# define STDC99 -# endif -# endif -#endif -#if !defined(STDC) && (defined(__STDC__) || defined(__cplusplus)) -# define STDC -#endif -#if !defined(STDC) && (defined(__GNUC__) || defined(__BORLANDC__)) -# define STDC -#endif -#if !defined(STDC) && (defined(MSDOS) || defined(WINDOWS) || defined(WIN32)) -# define STDC -#endif -#if !defined(STDC) && (defined(OS2) || defined(__HOS_AIX__)) -# define STDC -#endif - -#if defined(__OS400__) && !defined(STDC) /* iSeries (formerly AS/400). */ -# define STDC -#endif - -#ifndef STDC -# ifndef const /* cannot use !defined(STDC) && !defined(const) on Mac */ -# define const /* note: need a more gentle solution here */ -# endif -#endif - -/* Some Mac compilers merge all .h files incorrectly: */ -#if defined(__MWERKS__)||defined(applec)||defined(THINK_C)||defined(__SC__) -# define NO_DUMMY_DECL +/* Jeez, don't complain about non-prototype + * forms, we didn't write zlib */ +#if defined(_MSC_VER) +# pragma warning( disable : 4131 ) #endif /* Maximum value for memLevel in deflateInit2 */ -#ifndef MAX_MEM_LEVEL -# ifdef MAXSEG_64K -# define MAX_MEM_LEVEL 8 -# else -# define MAX_MEM_LEVEL 9 -# endif -#endif +#define MAX_MEM_LEVEL 9 /* Maximum value for windowBits in deflateInit2 and inflateInit2. * WARNING: reducing MAX_WBITS makes minigzip unable to extract .gz files * created by gzip. (Files created by minigzip can still be extracted by * gzip.) */ -#ifndef MAX_WBITS -# define MAX_WBITS 15 /* 32K LZ77 window */ -#endif - -/* The memory requirements for deflate are (in bytes): - (1 << (windowBits+2)) + (1 << (memLevel+9)) - that is: 128K for windowBits=15 + 128K for memLevel = 8 (default values) - plus a few kilobytes for small objects. For example, if you want to reduce - the default memory requirements from 256K to 128K, compile with - make CFLAGS="-O -DMAX_WBITS=14 -DMAX_MEM_LEVEL=7" - Of course this will generally degrade compression (there's no free lunch). - - The memory requirements for inflate are (in bytes) 1 << windowBits - that is, 32K for windowBits=15 (default value) plus a few kilobytes - for small objects. -*/ - - /* Type declarations */ - -#ifndef OF /* function prototypes */ -# ifdef STDC -# define OF(args) args -# else -# define OF(args) () -# endif -#endif - -/* The following definitions for FAR are needed only for MSDOS mixed - * model programming (small or medium model with some far allocations). - * This was tested only with MSC; for other MSDOS compilers you may have - * to define NO_MEMCPY in zutil.h. If you don't need the mixed model, - * just define FAR to be empty. - */ -#ifdef SYS16BIT -# if defined(M_I86SM) || defined(M_I86MM) - /* MSC small or medium model */ -# define SMALL_MEDIUM -# ifdef _MSC_VER -# define FAR _far -# else -# define FAR far -# endif -# endif -# if (defined(__SMALL__) || defined(__MEDIUM__)) - /* Turbo C small or medium model */ -# define SMALL_MEDIUM -# ifdef __BORLANDC__ -# define FAR _far -# else -# define FAR far -# endif -# endif -#endif - -#if defined(WINDOWS) || defined(WIN32) - /* If building or using zlib as a DLL, define ZLIB_DLL. - * This is not mandatory, but it offers a little performance increase. - */ -# ifdef ZLIB_DLL -# if defined(WIN32) && (!defined(__BORLANDC__) || (__BORLANDC__ >= 0x500)) -# ifdef ZLIB_INTERNAL -# define ZEXTERN extern __declspec(dllexport) -# else -# define ZEXTERN extern __declspec(dllimport) -# endif -# endif -# endif /* ZLIB_DLL */ - /* If building or using zlib with the WINAPI/WINAPIV calling convention, - * define ZLIB_WINAPI. - * Caution: the standard ZLIB1.DLL is NOT compiled using ZLIB_WINAPI. - */ -# ifdef ZLIB_WINAPI -# ifdef FAR -# undef FAR -# endif -# include - /* No need for _export, use ZLIB.DEF instead. */ - /* For complete Windows compatibility, use WINAPI, not __stdcall. */ -# define ZEXPORT WINAPI -# ifdef WIN32 -# define ZEXPORTVA WINAPIV -# else -# define ZEXPORTVA FAR CDECL -# endif -# endif -#endif - -#if defined (__BEOS__) -# ifdef ZLIB_DLL -# ifdef ZLIB_INTERNAL -# define ZEXPORT __declspec(dllexport) -# define ZEXPORTVA __declspec(dllexport) -# else -# define ZEXPORT __declspec(dllimport) -# define ZEXPORTVA __declspec(dllimport) -# endif -# endif -#endif - -#ifndef ZEXTERN -# define ZEXTERN extern -#endif -#ifndef ZEXPORT -# define ZEXPORT -#endif -#ifndef ZEXPORTVA -# define ZEXPORTVA -#endif +#define MAX_WBITS 15 /* 32K LZ77 window */ +#define ZEXTERN extern +#define ZEXPORT +#define ZEXPORTVA #ifndef FAR -# define FAR +# define FAR #endif +#define OF(args) args -#if !defined(__MACTYPES__) typedef unsigned char Byte; /* 8 bits */ -#endif typedef unsigned int uInt; /* 16 bits or more */ typedef unsigned long uLong; /* 32 bits or more */ -#ifdef SMALL_MEDIUM - /* Borland C/C++ and some old MSC versions ignore FAR inside typedef */ -# define Bytef Byte FAR -#else - typedef Byte FAR Bytef; -#endif +typedef Byte FAR Bytef; typedef char FAR charf; typedef int FAR intf; typedef uInt FAR uIntf; typedef uLong FAR uLongf; -#ifdef STDC - typedef void const *voidpc; - typedef void FAR *voidpf; - typedef void *voidp; -#else - typedef Byte const *voidpc; - typedef Byte FAR *voidpf; - typedef Byte *voidp; -#endif +typedef void const *voidpc; +typedef void FAR *voidpf; +typedef void *voidp; -#if 1 /* was set to #if 1 by ./configure */ -# define Z_HAVE_UNISTD_H -#endif - -#ifdef STDC -# include /* for off_t */ -#endif - -/* a little trick to accommodate both "#define _LARGEFILE64_SOURCE" and - * "#define _LARGEFILE64_SOURCE 1" as requesting 64-bit operations, (even - * though the former does not conform to the LFS document), but considering - * both "#undef _LARGEFILE64_SOURCE" and "#define _LARGEFILE64_SOURCE 0" as - * equivalently requesting no 64-bit operations - */ -#if -_LARGEFILE64_SOURCE - -1 == 1 -# undef _LARGEFILE64_SOURCE -#endif - -#if defined(Z_HAVE_UNISTD_H) || defined(_LARGEFILE64_SOURCE) -# include /* for SEEK_* and off_t */ -# ifdef VMS -# include /* for off_t */ -# endif -# ifndef z_off_t -# define z_off_t off_t -# endif -#endif - -#ifndef SEEK_SET -# define SEEK_SET 0 /* Seek from beginning of file. */ -# define SEEK_CUR 1 /* Seek from current position. */ -# define SEEK_END 2 /* Set file pointer to EOF plus "offset" */ -#endif - -#ifndef z_off_t -# define z_off_t long -#endif - -#if defined(_LARGEFILE64_SOURCE) && _LFS64_LARGEFILE-0 -# define z_off64_t off64_t -#else -# define z_off64_t z_off_t -#endif - -#if defined(__OS400__) -# define NO_vsnprintf -#endif - -#if defined(__MVS__) -# define NO_vsnprintf -#endif - -/* MVS linker does not support external names larger than 8 bytes */ -#if defined(__MVS__) - #pragma map(deflateInit_,"DEIN") - #pragma map(deflateInit2_,"DEIN2") - #pragma map(deflateEnd,"DEEND") - #pragma map(deflateBound,"DEBND") - #pragma map(inflateInit_,"ININ") - #pragma map(inflateInit2_,"ININ2") - #pragma map(inflateEnd,"INEND") - #pragma map(inflateSync,"INSY") - #pragma map(inflateSetDictionary,"INSEDI") - #pragma map(compressBound,"CMBND") - #pragma map(inflate_table,"INTABL") - #pragma map(inflate_fast,"INFA") - #pragma map(inflate_copyright,"INCOPY") -#endif +#define z_off_t git_off_t +#define z_off64_t z_off_t #endif /* ZCONF_H */ From 7ad96e51ca81974c417914edbc81a63e390c4301 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 15 Mar 2011 05:38:50 +0200 Subject: [PATCH 15/81] Remove duplicate refs in `git_reference_listall` --- src/refs.c | 29 ++++++++++++++++++----------- tests/t10-refs.c | 6 +++++- 2 files changed, 23 insertions(+), 12 deletions(-) diff --git a/src/refs.c b/src/refs.c index b9d25a769..cb11159fb 100644 --- a/src/refs.c +++ b/src/refs.c @@ -546,6 +546,7 @@ cleanup: struct dirent_list_data { git_vector ref_list; + git_repository *repo; size_t repo_path_len; unsigned int list_flags; }; @@ -553,16 +554,18 @@ struct dirent_list_data { static int _dirent_loose_listall(void *_data, char *full_path) { struct dirent_list_data *data = (struct dirent_list_data *)_data; - char *file_path; + char *file_path = full_path + data->repo_path_len; if (gitfo_isdir(full_path) == GIT_SUCCESS) return gitfo_dirent(full_path, GIT_PATH_MAX, _dirent_loose_listall, _data); + /* do not add twice a reference that exists already in the packfile */ + if (git_hashtable_lookup(data->repo->references.packfile, file_path) != NULL) + return GIT_SUCCESS; + if ((data->list_flags & loose_guess_rtype(full_path)) == 0) return GIT_SUCCESS; /* we are filtering out this reference */ - file_path = full_path + data->repo_path_len; - return git_vector_insert(&data->ref_list, git__strdup(file_path)); } @@ -1336,15 +1339,9 @@ int git_reference_listall(git_strarray *array, git_repository *repo, unsigned in git_vector_init(&data.ref_list, 8, NULL); data.repo_path_len = strlen(repo->path_repository); data.list_flags = list_flags; + data.repo = repo; - git__joinpath(refs_path, repo->path_repository, GIT_REFS_DIR); - error = gitfo_dirent(refs_path, GIT_PATH_MAX, _dirent_loose_listall, &data); - - if (error < GIT_SUCCESS) { - git_vector_free(&data.ref_list); - return error; - } - + /* list all the packed references first */ if (list_flags & GIT_REF_PACKED) { const char *ref_name; void *_unused; @@ -1359,6 +1356,16 @@ int git_reference_listall(git_strarray *array, git_repository *repo, unsigned in ); } + /* now list the loose references, trying not to + * duplicate the ref names already in the packed-refs file */ + git__joinpath(refs_path, repo->path_repository, GIT_REFS_DIR); + error = gitfo_dirent(refs_path, GIT_PATH_MAX, _dirent_loose_listall, &data); + + if (error < GIT_SUCCESS) { + git_vector_free(&data.ref_list); + return error; + } + array->strings = (char **)data.ref_list.contents; array->count = data.ref_list.length; return GIT_SUCCESS; diff --git a/tests/t10-refs.c b/tests/t10-refs.c index c70fb69ce..2c4c8a2dd 100644 --- a/tests/t10-refs.c +++ b/tests/t10-refs.c @@ -716,7 +716,11 @@ BEGIN_TEST(list0, "try to list all the references in our test repo") must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); must_pass(git_reference_listall(&ref_list, repo, GIT_REF_LISTALL)); - must_be_true(ref_list.count == 8); /* 8 refs in total if we include the packed ones */ + + /* We have exactly 7 refs in total if we include the packed ones: + * there is a reference that exists both in the packfile and as + * loose, but we only list it once */ + must_be_true(ref_list.count == 7); git_strarray_free(&ref_list); git_repository_free(repo); From 434bedcda0a7edb4fcc2dda6eb3df0310c2d0774 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 15 Mar 2011 19:46:38 +0200 Subject: [PATCH 16/81] Fix compilation warnings in ZLib (MSVC) Yes, we are changing the Zlib code. This is dangerous and uncool. Fortunately, these are just some implicit casts. --- deps/zlib/deflate.h | 6 +++--- deps/zlib/trees.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/deps/zlib/deflate.h b/deps/zlib/deflate.h index cbf0d1ea5..d7d26f8a9 100644 --- a/deps/zlib/deflate.h +++ b/deps/zlib/deflate.h @@ -317,15 +317,15 @@ void ZLIB_INTERNAL _tr_stored_block OF((deflate_state *s, charf *buf, #endif # define _tr_tally_lit(s, c, flush) \ - { uch cc = (c); \ + { uch cc = (uch)(c); \ s->d_buf[s->last_lit] = 0; \ s->l_buf[s->last_lit++] = cc; \ s->dyn_ltree[cc].Freq++; \ flush = (s->last_lit == s->lit_bufsize-1); \ } # define _tr_tally_dist(s, distance, length, flush) \ - { uch len = (length); \ - ush dist = (distance); \ + { uch len = (uch)(length); \ + ush dist = (ush)(distance); \ s->d_buf[s->last_lit] = dist; \ s->l_buf[s->last_lit++] = len; \ dist--; \ diff --git a/deps/zlib/trees.c b/deps/zlib/trees.c index 56e9bb1c1..3e9a138c7 100644 --- a/deps/zlib/trees.c +++ b/deps/zlib/trees.c @@ -605,7 +605,7 @@ local void gen_codes (tree, max_code, bl_count) int len = tree[n].Len; if (len == 0) continue; /* Now reverse the bits */ - tree[n].Code = bi_reverse(next_code[len]++, len); + tree[n].Code = (ush)bi_reverse(next_code[len]++, len); Tracecv(tree != static_ltree, (stderr,"\nn %3d %c l %2d c %4x (%x) ", n, (isgraph(n) ? n : ' '), len, tree[n].Code, next_code[len]-1)); @@ -729,7 +729,7 @@ local void scan_tree (s, tree, max_code) if (++count < max_count && curlen == nextlen) { continue; } else if (count < min_count) { - s->bl_tree[curlen].Freq += count; + s->bl_tree[curlen].Freq += (ush)count; } else if (curlen != 0) { if (curlen != prevlen) s->bl_tree[curlen].Freq++; s->bl_tree[REP_3_6].Freq++; From d40d30cb76ddcc9935471af0fb6fa6b2a66efc4c Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 15 Mar 2011 19:54:45 +0200 Subject: [PATCH 17/81] Debug assertion when using an initialized hashtable --- src/hashtable.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/hashtable.c b/src/hashtable.c index c36d8a8e6..ee6d3a461 100644 --- a/src/hashtable.c +++ b/src/hashtable.c @@ -184,6 +184,8 @@ int git_hashtable_insert2(git_hashtable *self, const void *key, void *value, voi int hash_id; git_hashtable_node *node; + assert(self && self->nodes); + *old_value = NULL; for (hash_id = 0; hash_id < GIT_HASHTABLE_HASHES; ++hash_id) { @@ -218,6 +220,8 @@ void *git_hashtable_lookup(git_hashtable *self, const void *key) int hash_id; git_hashtable_node *node; + assert(self && self->nodes); + for (hash_id = 0; hash_id < GIT_HASHTABLE_HASHES; ++hash_id) { node = node_with_hash(self, key, hash_id); if (node->key && self->key_equal(key, node->key) == 0) @@ -232,6 +236,8 @@ int git_hashtable_remove(git_hashtable *self, const void *key) int hash_id; git_hashtable_node *node; + assert(self && self->nodes); + for (hash_id = 0; hash_id < GIT_HASHTABLE_HASHES; ++hash_id) { node = node_with_hash(self, key, hash_id); if (node->key && self->key_equal(key, node->key) == 0) { From b5abb881a623b8b492e0375b8e9c8936079c39bb Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 15 Mar 2011 19:55:01 +0200 Subject: [PATCH 18/81] Do not segfault when listing unpacked references --- src/refs.c | 3 ++- tests/t10-refs.c | 19 +++++++++++++++++++ 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/src/refs.c b/src/refs.c index cb11159fb..ea39608df 100644 --- a/src/refs.c +++ b/src/refs.c @@ -560,7 +560,8 @@ static int _dirent_loose_listall(void *_data, char *full_path) return gitfo_dirent(full_path, GIT_PATH_MAX, _dirent_loose_listall, _data); /* do not add twice a reference that exists already in the packfile */ - if (git_hashtable_lookup(data->repo->references.packfile, file_path) != NULL) + if ((data->list_flags & GIT_REF_PACKED) != 0 && + git_hashtable_lookup(data->repo->references.packfile, file_path) != NULL) return GIT_SUCCESS; if ((data->list_flags & loose_guess_rtype(full_path)) == 0) diff --git a/tests/t10-refs.c b/tests/t10-refs.c index 2c4c8a2dd..565d636ba 100644 --- a/tests/t10-refs.c +++ b/tests/t10-refs.c @@ -717,6 +717,12 @@ BEGIN_TEST(list0, "try to list all the references in our test repo") must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); must_pass(git_reference_listall(&ref_list, repo, GIT_REF_LISTALL)); + /*{ + unsigned short i; + for (i = 0; i < ref_list.count; ++i) + printf("# %s\n", ref_list.strings[i]); + }*/ + /* We have exactly 7 refs in total if we include the packed ones: * there is a reference that exists both in the packfile and as * loose, but we only list it once */ @@ -726,6 +732,18 @@ BEGIN_TEST(list0, "try to list all the references in our test repo") git_repository_free(repo); END_TEST +BEGIN_TEST(list1, "try to list only the symbolic references") + git_repository *repo; + git_strarray ref_list; + + must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); + must_pass(git_reference_listall(&ref_list, repo, GIT_REF_SYMBOLIC)); + must_be_true(ref_list.count == 0); /* no symrefs in the test repo */ + + git_strarray_free(&ref_list); + git_repository_free(repo); +END_TEST + BEGIN_SUITE(refs) ADD_TEST(readtag0); @@ -758,4 +776,5 @@ BEGIN_SUITE(refs) ADD_TEST(delete0); ADD_TEST(list0); + ADD_TEST(list1); END_SUITE From bbcc7ffc690c4065954102530447aef4e8ccf895 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 15 Mar 2011 21:04:41 +0200 Subject: [PATCH 19/81] Add proper threading support to libgit2 We now depend on libpthread on all Unix platforms (should be installed by default) and use a simple wrapper for Windows threads under Win32. Signed-off-by: Vicent Marti --- README.md | 10 +-- include/git2/thread-utils.h | 10 +-- src/common.h | 5 +- src/thread-utils.h | 117 ++++++----------------------------- src/win32/pthread.c | 120 ++++++++++++++++++++++++++++++++++++ src/win32/pthread.h | 67 ++++++++++++++++++++ tests/t00-core.c | 13 ---- wscript | 44 ++++++++----- 8 files changed, 244 insertions(+), 142 deletions(-) create mode 100644 src/win32/pthread.c create mode 100644 src/win32/pthread.h diff --git a/README.md b/README.md index c9eb6dc25..05378685b 100644 --- a/README.md +++ b/README.md @@ -34,15 +34,17 @@ libgit2 is already very usable. Building libgit2 - External dependencies ======================================== -libgit2 builds cleanly on most platforms without any external dependencies. However, the following libraries -may be used on some platforms. +libgit2 builds cleanly on most platforms without any external dependencies. +Under Unix-like systems, like Linux, *BSD and Mac OS X, libgit2 expects `pthreads` to be available; +they should be installed by default on all systems. Under Windows, libgit2 uses the native Windows API +for threading. + +Additionally, he following libraries may be used as replacement for built-in functionality: * LibSSL **(optional)** libgit2 can be built using the SHA1 implementation of LibSSL-Crypto, instead of the built-in custom implementations. Performance wise, they are quite similar. -* pthreads-w32 **(required for MinGW)** - Building libgit2 - Using waf ====================== diff --git a/include/git2/thread-utils.h b/include/git2/thread-utils.h index c45a76e95..ecb09bb53 100644 --- a/include/git2/thread-utils.h +++ b/include/git2/thread-utils.h @@ -32,7 +32,6 @@ */ #define GIT_HAS_TLS 1 -#define GIT_HAS_PTHREAD 1 #if defined(__APPLE__) && defined(__MACH__) # undef GIT_TLS @@ -47,7 +46,6 @@ #elif defined(__INTEL_COMPILER) # if defined(_WIN32) || defined(_WIN32_CE) # define GIT_TLS __declspec(thread) -# undef GIT_HAS_PTHREAD # else # define GIT_TLS __thread # endif @@ -56,11 +54,9 @@ defined(_WIN32_CE) || \ defined(__BORLANDC__) # define GIT_TLS __declspec(thread) -# undef GIT_HAS_PTHREAD #else # undef GIT_HAS_TLS -# undef GIT_HAS_PTHREAD # define GIT_TLS /* nothing: tls vars are thread-global */ #endif @@ -71,10 +67,6 @@ # define GIT_TLS #endif -#ifdef GIT_HAS_PTHREAD -# define GIT_THREADS 1 -#else -# undef GIT_THREADS -#endif +#define GIT_THREADS 1 #endif /* INCLUDE_git_thread_utils_h__ */ diff --git a/src/common.h b/src/common.h index 1ca00471b..723085a2b 100644 --- a/src/common.h +++ b/src/common.h @@ -11,9 +11,6 @@ #include "git2/thread-utils.h" #include "cc-compat.h" -#ifdef GIT_HAS_PTHREAD -# include -#endif #ifdef GIT_HAVE_INTTYPES_H # include #endif @@ -34,6 +31,7 @@ # include # include "msvc-compat.h" # include "mingw-compat.h" +# include "win32/pthread.h" # define snprintf _snprintf @@ -43,6 +41,7 @@ typedef SSIZE_T ssize_t; # include # include +# include #endif diff --git a/src/thread-utils.h b/src/thread-utils.h index 0029e4bc1..e8372e731 100644 --- a/src/thread-utils.h +++ b/src/thread-utils.h @@ -1,108 +1,27 @@ #ifndef INCLUDE_thread_utils_h__ #define INCLUDE_thread_utils_h__ -#if defined(GIT_HAS_PTHREAD) - typedef pthread_t git_thread; -# define git_thread_create(thread, attr, start_routine, arg) pthread_create(thread, attr, start_routine, arg) -# define git_thread_kill(thread) pthread_cancel(thread) -# define git_thread_exit(status) pthread_exit(status) -# define git_thread_join(id, status) pthread_join(id, status) - /* Pthreads Mutex */ - typedef pthread_mutex_t git_lck; -# define GITLCK_INIT PTHREAD_MUTEX_INITIALIZER -# define gitlck_init(a) pthread_mutex_init(a, NULL) -# define gitlck_lock(a) pthread_mutex_lock(a) -# define gitlck_unlock(a) pthread_mutex_unlock(a) -# define gitlck_free(a) pthread_mutex_destroy(a) +#define git_thread pthread_t +#define git_thread_create(thread, attr, start_routine, arg) pthread_create(thread, attr, start_routine, arg) +#define git_thread_kill(thread) pthread_cancel(thread) +#define git_thread_exit(status) pthread_exit(status) +#define git_thread_join(id, status) pthread_join(id, status) - /* Pthreads condition vars */ - typedef pthread_cond_t git_cnd; -# define GITCND_INIT PTHREAD_COND_INITIALIZER -# define gitcnd_init(c, a) pthread_cond_init(c, a) -# define gitcnd_free(c) pthread_cond_destroy(c) -# define gitcnd_wait(c, l) pthread_cond_wait(c, l) -# define gitcnd_signal(c) pthread_cond_signal(c) -# define gitcnd_broadcast(c) pthread_cond_broadcast(c) +/* Pthreads Mutex */ +#define git_mutex pthread_mutex_t +#define git_mutex_init(a) pthread_mutex_init(a, NULL) +#define git_mutex_lock(a) pthread_mutex_lock(a) +#define git_mutex_unlock(a) pthread_mutex_unlock(a) +#define git_mutex_free(a) pthread_mutex_destroy(a) -# if defined(GIT_HAS_ASM_ATOMIC) -# include - typedef atomic_t git_refcnt; -# define gitrc_init(a, v) atomic_set(a, v) -# define gitrc_inc(a) atomic_inc_return(a) -# define gitrc_dec(a) atomic_dec_and_test(a) -# define gitrc_free(a) (void)0 -# elif defined(GIT_WIN32) - typedef long git_refcnt; -# define gitrc_init(a, v) (*a = v) -# define gitrc_inc(a) (InterlockedIncrement(a)) -# define gitrc_dec(a) (!InterlockedDecrement(a)) -# define gitrc_free(a) (void)0 -# else - typedef struct { git_lck lock; int counter; } git_refcnt; - - /** Initialize to 0. No memory barrier is issued. */ - GIT_INLINE(void) gitrc_init(git_refcnt *p, int value) - { - gitlck_init(&p->lock); - p->counter = value; - } - - /** - * Increment. - * - * Atomically increments @p by 1. A memory barrier is also - * issued before and after the operation. - * - * @param p pointer of type git_refcnt - */ - GIT_INLINE(void) gitrc_inc(git_refcnt *p) - { - gitlck_lock(&p->lock); - p->counter++; - gitlck_unlock(&p->lock); - } - - /** - * Decrement and test. - * - * Atomically decrements @p by 1 and returns true if the - * result is 0, or false for all other cases. A memory - * barrier is also issued before and after the operation. - * - * @param p pointer of type git_refcnt - */ - GIT_INLINE(int) gitrc_dec(git_refcnt *p) - { - int c; - gitlck_lock(&p->lock); - c = --p->counter; - gitlck_unlock(&p->lock); - return !c; - } - - /** Free any resources associated with the counter. */ -# define gitrc_free(p) gitlck_free(&(p)->lock) -# endif - -#elif defined(GIT_THREADS) -# error GIT_THREADS but no git_lck implementation - -#else - /* no threads support */ - typedef struct { int dummy; } git_lck; -# define GIT_MUTEX_INIT {} -# define gitlck_init(a) (void)0 -# define gitlck_lock(a) (void)0 -# define gitlck_unlock(a) (void)0 -# define gitlck_free(a) (void)0 - - typedef struct { int counter; } git_refcnt; -# define gitrc_init(a,v) ((a)->counter = v) -# define gitrc_inc(a) ((a)->counter++) -# define gitrc_dec(a) (--(a)->counter == 0) -# define gitrc_free(a) (void)0 -#endif +/* Pthreads condition vars */ +#define git_cond pthread_cond_t +#define git_cond_init(c, a) pthread_cond_init(c, a) +#define git_cond_free(c) pthread_cond_destroy(c) +#define git_cond_wait(c, l) pthread_cond_wait(c, l) +#define git_cond_signal(c) pthread_cond_signal(c) +#define git_cond_broadcast(c) pthread_cond_broadcast(c) extern int git_online_cpus(void); diff --git a/src/win32/pthread.c b/src/win32/pthread.c new file mode 100644 index 000000000..fffff81df --- /dev/null +++ b/src/win32/pthread.c @@ -0,0 +1,120 @@ +/* + * This file is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License, version 2, + * as published by the Free Software Foundation. + * + * In addition to the permissions in the GNU General Public License, + * the authors give you unlimited permission to link the compiled + * version of this file into combinations with other programs, + * and to distribute those combinations without any restriction + * coming from the use of this file. (The General Public License + * restrictions do apply in other respects; for example, they cover + * modification of the file, and distribution when not linked into + * a combined executable.) + * + * This file is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; see the file COPYING. If not, write to + * the Free Software Foundation, 51 Franklin Street, Fifth Floor, + * Boston, MA 02110-1301, USA. + * + * Original code by Ramiro Polla (Public Domain) + */ + +#include "pthread.h" + +int pthread_create(pthread_t *GIT_RESTRICT thread, + const pthread_attr_t *GIT_RESTRICT GIT_UNUSED(attr), + void *(*start_routine)(void*), void *GIT_RESTRICT arg) +{ + GIT_UNUSED_ARG(attr); + *thread = (pthread_t) CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)start_routine, arg, 0, NULL); + return *thread ? GIT_SUCCESS : GIT_EOSERR; +} + +int pthread_cond_signal(pthread_cond_t *cond) +{ + WakeConditionVariable(cond); + return 0; +} + +int pthread_cond_wait(pthread_cond_t *GIT_RESTRICT cond, + pthread_mutex_t *GIT_RESTRICT mutex) +{ + int ret; + ret = SleepConditionVariableCS(cond, mutex, INFINITE); + return -(!ret); +} + +int pthread_mutex_lock(pthread_mutex_t *mutex) +{ + EnterCriticalSection(mutex); + return 0; +} + +int pthread_mutex_unlock(pthread_mutex_t *mutex) +{ + LeaveCriticalSection(mutex); + return 0; +} + +int pthread_join(pthread_t thread, void **value_ptr) +{ + int ret; + ret = WaitForSingleObject(thread, INFINITE); + if (ret && value_ptr) + GetExitCodeThread(thread, (void*) value_ptr); + return -(!!ret); +} + +int pthread_cond_broadcast(pthread_cond_t *cond) +{ + WakeAllConditionVariable(cond); + return 0; +} + +int pthread_mutex_destroy(pthread_mutex_t *mutex) +{ + int ret; + ret = CloseHandle(mutex); + return -(!ret); +} + +int pthread_cond_destroy(pthread_cond_t *GIT_UNUSED(cond)) +{ + GIT_UNUSED_ARG(cond); + return 0; +} + +int pthread_cond_init(pthread_cond_t *GIT_RESTRICT cond, + const pthread_condattr_t *GIT_RESTRICT GIT_UNUSED(condattr)) +{ + GIT_UNUSED_ARG(condattr); + InitializeConditionVariable(cond); + return 0; +} + +int pthread_mutex_init(pthread_mutex_t *GIT_RESTRICT mutex, + const pthread_mutexattr_t *GIT_RESTRICT GIT_UNUSED(mutexattr)) +{ + GIT_UNUSED_ARG(mutexattr); + InitializeCriticalSection(mutex); + return 0; +} + +int pthread_num_processors_np(void) +{ + DWORD_PTR p, s; + int n = 0; + + if (GetProcessAffinityMask(GetCurrentProcess(), &p, &s)) + for (; p; p >>= 1) + n += p&1; + + return n ? n : 1; +} + diff --git a/src/win32/pthread.h b/src/win32/pthread.h new file mode 100644 index 000000000..ff694e303 --- /dev/null +++ b/src/win32/pthread.h @@ -0,0 +1,67 @@ +/* + * This file is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License, version 2, + * as published by the Free Software Foundation. + * + * In addition to the permissions in the GNU General Public License, + * the authors give you unlimited permission to link the compiled + * version of this file into combinations with other programs, + * and to distribute those combinations without any restriction + * coming from the use of this file. (The General Public License + * restrictions do apply in other respects; for example, they cover + * modification of the file, and distribution when not linked into + * a combined executable.) + * + * This file is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; see the file COPYING. If not, write to + * the Free Software Foundation, 51 Franklin Street, Fifth Floor, + * Boston, MA 02110-1301, USA. + * + * Original code by Ramiro Polla (Public Domain) + */ + +#ifndef GIT_PTHREAD_H +#define GIT_PTHREAD_H + +#include "../common.h" + +#if defined (_MSC_VER) +# define GIT_RESTRICT __restrict +#else +# define GIT_RESTRICT __restrict__ +#endif + +typedef int pthread_mutexattr_t; +typedef int pthread_condattr_t; +typedef int pthread_attr_t; +typedef CRITICAL_SECTION pthread_mutex_t; +typedef CONDITION_VARIABLE pthread_cond_t; +typedef HANDLE pthread_t; + +#define PTHREAD_MUTEX_INITIALIZER {(void*)-1}; + +int pthread_create(pthread_t *GIT_RESTRICT, + const pthread_attr_t *GIT_RESTRICT, + void *(*start_routine)(void*), void *__restrict); + +int pthread_join(pthread_t, void **); + +int pthread_mutex_init(pthread_mutex_t *GIT_RESTRICT, const pthread_mutexattr_t *GIT_RESTRICT); +int pthread_mutex_destroy(pthread_mutex_t *); +int pthread_mutex_lock(pthread_mutex_t *); +int pthread_mutex_unlock(pthread_mutex_t *); + +int pthread_cond_init(pthread_cond_t *GIT_RESTRICT, const pthread_condattr_t *GIT_RESTRICT); +int pthread_cond_destroy(pthread_cond_t *); +int pthread_cond_broadcast(pthread_cond_t *); +int pthread_cond_signal(pthread_cond_t *); +int pthread_cond_wait(pthread_cond_t *GIT_RESTRICT, pthread_mutex_t *GIT_RESTRICT); + +int pthread_num_processors_np(void); + +#endif \ No newline at end of file diff --git a/tests/t00-core.c b/tests/t00-core.c index 1f6c06a95..08e42ecf4 100644 --- a/tests/t00-core.c +++ b/tests/t00-core.c @@ -27,17 +27,6 @@ #include "vector.h" #include "fileops.h" -BEGIN_TEST(refcnt0, "increment refcount twice, decrement twice") - git_refcnt p; - - gitrc_init(&p, 0); - gitrc_inc(&p); - gitrc_inc(&p); - must_be_true(!gitrc_dec(&p)); - must_be_true(gitrc_dec(&p)); - gitrc_free(&p); -END_TEST - BEGIN_TEST(string0, "compare prefixes") must_be_true(git__prefixcmp("", "") == 0); must_be_true(git__prefixcmp("a", "") == 0); @@ -626,8 +615,6 @@ END_TEST BEGIN_SUITE(core) - ADD_TEST(refcnt0); - ADD_TEST(string0); ADD_TEST(string1); diff --git a/wscript b/wscript index d791d932d..198b4760b 100644 --- a/wscript +++ b/wscript @@ -5,10 +5,12 @@ from waflib.Build import BuildContext, CleanContext, \ # Unix flags CFLAGS_UNIX = ["-O2", "-Wall", "-Wextra"] CFLAGS_UNIX_DBG = ['-g'] +CFLAGS_UNIX_PED = ['-pedantic', '-Werror'] # Windows MSVC flags -CFLAGS_WIN32_COMMON = ['/TC', '/W4', '/WX', '/nologo', '/Zi'] +CFLAGS_WIN32_COMMON = ['/TC', '/W4', '/nologo', '/Zi'] CFLAGS_WIN32_RELEASE = ['/O2', '/MD'] +CFLAGS_WIN32_PED = ['/Wx'] # Note: /RTC* cannot be used with optimization on. CFLAGS_WIN32_DBG = ['/Od', '/RTC1', '/RTCc', '/DEBUG', '/MDd'] @@ -31,9 +33,10 @@ PPC optimized version (ppc) or the SHA1 functions from OpenSSL (openssl)") help='Select target architecture (ia64, x64, x86, x86_amd64, x86_ia64)') opt.add_option('--without-sqlite', action='store_false', default=True, dest='use_sqlite', help='Disable sqlite support') + opt.add_option('--strict', action='store_true', default=False, + help='Max warning level; treat warnings as errors') def configure(conf): - # load the MSVC configuration flags if conf.options.msvc: conf.env['MSVC_VERSIONS'] = ['msvc ' + conf.options.msvc] @@ -43,25 +46,38 @@ def configure(conf): # default configuration for C programs conf.load('compiler_c') - dbg = conf.options.debug + debug = conf.options.debug + pedantic = conf.options.strict - conf.env.CFLAGS = CFLAGS_UNIX + (CFLAGS_UNIX_DBG if dbg else []) + if conf.env.CC_NAME == 'msvc': + conf.env.CFLAGS = CFLAGS_WIN32_COMMON + conf.env.LINKFLAGS = CFLAGS_WIN32_L + if debug: + conf.env.CFLAGS += CFLAGS_WIN32_DEBUG + conf.env.LINKFLAGS += CFLAGS_WIN32_L_DBG + else: + conf.env.CFLAGS += CFLAGS_WIN32_RELEASE + if pedantic: + conf.ENV.CFLAGS += CFLAGS_WIN32_PED + + elif conf.env.CC_NAME == 'gcc': + conf.env.CFLAGS = CFLAGS_UNIX + if debug: + conf.env.CFLAGS += CFLAGS_UNIX_DBG + if pedantic: + conf.env.CFLAGS += CFLAGS_UNIX_PED + + + # Win32 Platform: MinGW and MSVC if conf.env.DEST_OS == 'win32': conf.env.PLATFORM = 'win32' + conf.env.DEFINES += ['WIN32', '_DEBUG', '_LIB'] - if conf.env.CC_NAME == 'msvc': - conf.env.CFLAGS = CFLAGS_WIN32_COMMON + \ - (CFLAGS_WIN32_DBG if dbg else CFLAGS_WIN32_RELEASE) - conf.env.LINKFLAGS += CFLAGS_WIN32_L + \ - (CFLAGS_WIN32_L_DBG if dbg else []) - conf.env.DEFINES += ['WIN32', '_DEBUG', '_LIB'] - - elif conf.env.CC_NAME == 'gcc': - conf.check_cc(lib='pthread', uselib_store='pthread') - + # Unix-like platforms: Linux, Darwin, Cygwin, *BSD, etc else: conf.env.PLATFORM = 'unix' + conf.check_cc(lib='pthread', uselib_store='pthread') # Do not build ZLib with GZIP support conf.env.DEFINES += ['NO_GZIP'] From f73b09cd9a36e1d02fe5b6ad33053e60ac88bc7a Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 15 Mar 2011 21:19:02 +0200 Subject: [PATCH 20/81] Revert changes in wscript file --- wscript | 40 +++++++++++----------------------------- 1 file changed, 11 insertions(+), 29 deletions(-) diff --git a/wscript b/wscript index 198b4760b..6d5579fef 100644 --- a/wscript +++ b/wscript @@ -5,12 +5,10 @@ from waflib.Build import BuildContext, CleanContext, \ # Unix flags CFLAGS_UNIX = ["-O2", "-Wall", "-Wextra"] CFLAGS_UNIX_DBG = ['-g'] -CFLAGS_UNIX_PED = ['-pedantic', '-Werror'] # Windows MSVC flags -CFLAGS_WIN32_COMMON = ['/TC', '/W4', '/nologo', '/Zi'] +CFLAGS_WIN32_COMMON = ['/TC', '/W4', '/WX', '/nologo', '/Zi'] CFLAGS_WIN32_RELEASE = ['/O2', '/MD'] -CFLAGS_WIN32_PED = ['/Wx'] # Note: /RTC* cannot be used with optimization on. CFLAGS_WIN32_DBG = ['/Od', '/RTC1', '/RTCc', '/DEBUG', '/MDd'] @@ -33,10 +31,9 @@ PPC optimized version (ppc) or the SHA1 functions from OpenSSL (openssl)") help='Select target architecture (ia64, x64, x86, x86_amd64, x86_ia64)') opt.add_option('--without-sqlite', action='store_false', default=True, dest='use_sqlite', help='Disable sqlite support') - opt.add_option('--strict', action='store_true', default=False, - help='Max warning level; treat warnings as errors') def configure(conf): + # load the MSVC configuration flags if conf.options.msvc: conf.env['MSVC_VERSIONS'] = ['msvc ' + conf.options.msvc] @@ -46,35 +43,20 @@ def configure(conf): # default configuration for C programs conf.load('compiler_c') - debug = conf.options.debug - pedantic = conf.options.strict + dbg = conf.options.debug - if conf.env.CC_NAME == 'msvc': - conf.env.CFLAGS = CFLAGS_WIN32_COMMON - conf.env.LINKFLAGS = CFLAGS_WIN32_L - if debug: - conf.env.CFLAGS += CFLAGS_WIN32_DEBUG - conf.env.LINKFLAGS += CFLAGS_WIN32_L_DBG - else: - conf.env.CFLAGS += CFLAGS_WIN32_RELEASE + conf.env.CFLAGS = CFLAGS_UNIX + (CFLAGS_UNIX_DBG if dbg else []) - if pedantic: - conf.ENV.CFLAGS += CFLAGS_WIN32_PED - - elif conf.env.CC_NAME == 'gcc': - conf.env.CFLAGS = CFLAGS_UNIX - if debug: - conf.env.CFLAGS += CFLAGS_UNIX_DBG - if pedantic: - conf.env.CFLAGS += CFLAGS_UNIX_PED - - - # Win32 Platform: MinGW and MSVC if conf.env.DEST_OS == 'win32': conf.env.PLATFORM = 'win32' - conf.env.DEFINES += ['WIN32', '_DEBUG', '_LIB'] - # Unix-like platforms: Linux, Darwin, Cygwin, *BSD, etc + if conf.env.CC_NAME == 'msvc': + conf.env.CFLAGS = CFLAGS_WIN32_COMMON + \ + (CFLAGS_WIN32_DBG if dbg else CFLAGS_WIN32_RELEASE) + conf.env.LINKFLAGS += CFLAGS_WIN32_L + \ + (CFLAGS_WIN32_L_DBG if dbg else []) + conf.env.DEFINES += ['WIN32', '_DEBUG', '_LIB'] + else: conf.env.PLATFORM = 'unix' conf.check_cc(lib='pthread', uselib_store='pthread') From 36b313296638f8ae349e400ba9a7f5bbfb92b96f Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 16 Mar 2011 01:04:17 +0200 Subject: [PATCH 21/81] Properly free commit a commit list in revwalk The commit list was not being properly free'd when a walk was stopped halfway through. --- src/revwalk.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/src/revwalk.c b/src/revwalk.c index edafbe73d..d17b44053 100644 --- a/src/revwalk.c +++ b/src/revwalk.c @@ -78,13 +78,17 @@ commit_list *commit_list_insert(commit_object *item, commit_list **list_p) return new_list; } -void commit_list_free(commit_list *list) +void commit_list_free(commit_list **list_p) { + commit_list *list = *list_p; + while (list) { commit_list *temp = list; list = temp->next; free(temp); } + + *list_p = NULL; } commit_object *commit_list_pop(commit_list **stack) @@ -561,9 +565,9 @@ void git_revwalk_reset(git_revwalk *walk) ); git_pqueue_free(&walk->iterator_time); - commit_list_free(walk->iterator_topo); - commit_list_free(walk->iterator_rand); - commit_list_free(walk->iterator_reverse); + commit_list_free(&walk->iterator_topo); + commit_list_free(&walk->iterator_rand); + commit_list_free(&walk->iterator_reverse); walk->walking = 0; } From 955f9ae9bd25ceb59775463e04c0276209387b67 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 16 Mar 2011 01:06:15 +0200 Subject: [PATCH 22/81] Export `git_strarray_free` instead of inlining That way non-C bindings can use it. --- include/git2/common.h | 9 +-------- src/util.c | 9 +++++++++ 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/include/git2/common.h b/include/git2/common.h index 11a08f897..f9045210c 100644 --- a/include/git2/common.h +++ b/include/git2/common.h @@ -165,14 +165,7 @@ typedef struct { size_t count; } git_strarray; -GIT_INLINE(void) git_strarray_free(git_strarray *array) -{ - size_t i; - for (i = 0; i < array->count; ++i) - free(array->strings[i]); - - free(array->strings); -} +GIT_EXTERN(void) git_strarray_free(git_strarray *array); /** @} */ GIT_END_DECL diff --git a/src/util.c b/src/util.c index c9a8e5fe9..bfc4f7b27 100644 --- a/src/util.c +++ b/src/util.c @@ -3,6 +3,15 @@ #include #include +void git_strarray_free(git_strarray *array) +{ + size_t i; + for (i = 0; i < array->count; ++i) + free(array->strings[i]); + + free(array->strings); +} + int git__fmt(char *buf, size_t buf_sz, const char *fmt, ...) { va_list va; From 54a1b36c257a490f13a26d92d4c8e5e3912d00cd Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 16 Mar 2011 01:07:06 +0200 Subject: [PATCH 23/81] Export `git_repository_gc` properly One of my brainfarts made me export it as `git_repository_close` instead of GC. Duh. --- include/git2/repository.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/git2/repository.h b/include/git2/repository.h index d5f6cf501..5eff53b29 100644 --- a/include/git2/repository.h +++ b/include/git2/repository.h @@ -167,7 +167,7 @@ GIT_EXTERN(void) git_repository_free(git_repository *repo); * * @param repo repository handle to collect. If NULL nothing occurs. */ -GIT_EXTERN(void) git_repository_close(git_repository *repo); +GIT_EXTERN(void) git_repository_gc(git_repository *repo); /** * Creates a new Git repository in the given folder. From 8613d4a930541dccdc948cb18e313551ec42cb0e Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 16 Mar 2011 01:10:40 +0200 Subject: [PATCH 24/81] Fix signature in `git_repository_gc` The method returns an int with the amount of objects free'd --- include/git2/repository.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/git2/repository.h b/include/git2/repository.h index 5eff53b29..317b367d2 100644 --- a/include/git2/repository.h +++ b/include/git2/repository.h @@ -167,7 +167,7 @@ GIT_EXTERN(void) git_repository_free(git_repository *repo); * * @param repo repository handle to collect. If NULL nothing occurs. */ -GIT_EXTERN(void) git_repository_gc(git_repository *repo); +GIT_EXTERN(int) git_repository_gc(git_repository *repo); /** * Creates a new Git repository in the given folder. From 36aaf1ff1a8540c364050f62f19f3021b6e121eb Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 16 Mar 2011 01:53:25 +0200 Subject: [PATCH 25/81] Change the Revwalk reset behavior to the old version The `reset` call now removes the pushed commits so we can reuse the revwalker. The API documentation has been updated with the details. --- include/git2/revwalk.h | 65 +++++++++++++-- src/pqueue.c | 4 + src/pqueue.h | 5 ++ src/revwalk.c | 179 +++++++++++++++++++---------------------- tests/t05-revwalk.c | 17 ++-- 5 files changed, 157 insertions(+), 113 deletions(-) diff --git a/include/git2/revwalk.h b/include/git2/revwalk.h index fdbbe236c..4b5ea8e06 100644 --- a/include/git2/revwalk.h +++ b/include/git2/revwalk.h @@ -70,6 +70,17 @@ GIT_BEGIN_DECL /** * Allocate a new revision walker to iterate through a repo. * + * This revision walker uses a custom memory pool and an internal + * commit cache, so it is relatively expensive to allocate. + * + * For maximum performance, this revision walker should be + * reused for different walks. + * + * This revision walker is *not* thread safe: it may only be + * used to walk a repository on a single thread; however, + * it is possible to have several revision walkers in + * several different threads walking the same repository. + * * @param walker pointer to the new revision walker * @param repo the repo to walk through * @return 0 on success; error code otherwise @@ -77,32 +88,67 @@ GIT_BEGIN_DECL GIT_EXTERN(int) git_revwalk_new(git_revwalk **walker, git_repository *repo); /** - * Reset the walking machinery for reuse. + * Reset the revision walker for reuse. + * + * This will clear all the pushed and hidden commits, and + * leave the walker in a blank state (just like at + * creation) ready to receive new commit pushes and + * start a new walk. + * + * The revision walk is automatically reset when a walk + * is over. + * * @param walker handle to reset. */ GIT_EXTERN(void) git_revwalk_reset(git_revwalk *walker); /** * Mark a commit to start traversal from. - * The commit object must belong to the repo which is being walked through. + * + * The given OID must belong to a commit on the walked + * repository. + * + * The given commit will be used as one of the roots + * when starting the revision walk. At least one commit + * must be pushed the repository before a walk can + * be started. * * @param walker the walker being used for the traversal. - * @param commit the commit to start from. + * @param oid the oid of the commit to start from. + * @return 0 on success; error code otherwise */ GIT_EXTERN(int) git_revwalk_push(git_revwalk *walk, const git_oid *oid); /** * Mark a commit (and its ancestors) uninteresting for the output. + * + * The given OID must belong to a commit on the walked + * repository. + * + * The resolved commit and all its parents will be hidden from the + * output on the revision walk. + * * @param walker the walker being used for the traversal. * @param commit the commit that will be ignored during the traversal + * @return 0 on success; error code otherwise */ GIT_EXTERN(int) git_revwalk_hide(git_revwalk *walk, const git_oid *oid); /** - * Get the next commit from the revision traversal. + * Get the next commit from the revision walk. * - * @param commit Pointer where to store the next commit + * The initial call to this method is *not* blocking when + * iterating through a repo with a time-sorting mode. + * + * Iterating with Topological or inverted modes makes the initial + * call blocking to preprocess the commit list, but this block should be + * mostly unnoticeable on most repositories (topological preprocessing + * times at 0.3s on the git.git repo). + * + * The revision walker is reset when the walk is over. + * + * @param oid Pointer where to store the oid of the next commit * @param walk the walker to pop the commit from. * @return GIT_SUCCESS if the next commit was found; * GIT_EREVWALKOVER if there are no commits left to iterate @@ -112,14 +158,17 @@ GIT_EXTERN(int) git_revwalk_next(git_oid *oid, git_revwalk *walk); /** * Change the sorting mode when iterating through the * repository's contents. + * * Changing the sorting mode resets the walker. + * * @param walk the walker being used for the traversal. - * @param sort_mode combination of GIT_RPSORT_XXX flags + * @param sort_mode combination of GIT_SORT_XXX flags */ -GIT_EXTERN(int) git_revwalk_sorting(git_revwalk *walk, unsigned int sort_mode); +GIT_EXTERN(void) git_revwalk_sorting(git_revwalk *walk, unsigned int sort_mode); /** - * Free a revwalk previously allocated. + * Free a revision walker previously allocated. + * * @param walk traversal handle to close. If NULL nothing occurs. */ GIT_EXTERN(void) git_revwalk_free(git_revwalk *walk); diff --git a/src/pqueue.c b/src/pqueue.c index 98152cb85..6307175e3 100644 --- a/src/pqueue.c +++ b/src/pqueue.c @@ -50,6 +50,10 @@ void git_pqueue_free(git_pqueue *q) q->d = NULL; } +void git_pqueue_clear(git_pqueue *q) +{ + q->size = 1; +} size_t git_pqueue_size(git_pqueue *q) { diff --git a/src/pqueue.h b/src/pqueue.h index 6db74661d..7a1394803 100644 --- a/src/pqueue.h +++ b/src/pqueue.h @@ -53,6 +53,11 @@ int git_pqueue_init(git_pqueue *q, size_t n, git_pqueue_cmp cmppri); */ void git_pqueue_free(git_pqueue *q); +/** + * clear all the elements in the queue + * @param q the queue + */ +void git_pqueue_clear(git_pqueue *q); /** * return the size of the queue. diff --git a/src/revwalk.c b/src/revwalk.c index d17b44053..a2fc092d5 100644 --- a/src/revwalk.c +++ b/src/revwalk.c @@ -52,7 +52,6 @@ struct git_revwalk { git_repository *repo; git_hashtable *commits; - git_vector pending; commit_list *iterator_topo; commit_list *iterator_rand; @@ -159,71 +158,6 @@ static commit_object **alloc_parents(commit_object *commit, size_t n_parents) return git__malloc(n_parents * sizeof(commit_object *)); } -int git_revwalk_new(git_revwalk **revwalk_out, git_repository *repo) -{ - git_revwalk *walk; - - walk = git__malloc(sizeof(git_revwalk)); - if (walk == NULL) - return GIT_ENOMEM; - - memset(walk, 0x0, sizeof(git_revwalk)); - - walk->commits = git_hashtable_alloc(64, - object_table_hash, - (git_hash_keyeq_ptr)git_oid_cmp); - - if (walk->commits == NULL) { - free(walk); - return GIT_ENOMEM; - } - - git_vector_init(&walk->pending, 8, NULL); - git_vector_init(&walk->memory_alloc, 8, NULL); - alloc_chunk(walk); - - walk->repo = repo; - - *revwalk_out = walk; - return GIT_SUCCESS; -} - -void git_revwalk_free(git_revwalk *walk) -{ - unsigned int i; - - if (walk == NULL) - return; - - git_revwalk_reset(walk); - git_hashtable_free(walk->commits); - git_vector_free(&walk->pending); - - for (i = 0; i < walk->memory_alloc.length; ++i) { - free(git_vector_get(&walk->memory_alloc, i)); - } - - git_vector_free(&walk->memory_alloc); - free(walk); -} - -git_repository *git_revwalk_repository(git_revwalk *walk) -{ - assert(walk); - return walk->repo; -} - -int git_revwalk_sorting(git_revwalk *walk, unsigned int sort_mode) -{ - assert(walk); - - if (walk->walking) - return GIT_EBUSY; - - walk->sorting = sort_mode; - git_revwalk_reset(walk); - return GIT_SUCCESS; -} static commit_object *commit_lookup(git_revwalk *walk, const git_oid *oid) { @@ -370,10 +304,9 @@ static int push_commit(git_revwalk *walk, const git_oid *oid, int uninteresting) if (commit == NULL) return GIT_ENOTFOUND; - if (uninteresting) - mark_uninteresting(commit); + commit->uninteresting = uninteresting; - return git_vector_insert(&walk->pending, commit); + return process_commit(walk, commit); } int git_revwalk_push(git_revwalk *walk, const git_oid *oid) @@ -472,31 +405,11 @@ static int revwalk_next_reverse(commit_object **object_out, git_revwalk *walk) static int prepare_walk(git_revwalk *walk) { - unsigned int i; int error; - - if (walk->sorting & GIT_SORT_TIME) { - if ((error = git_pqueue_init(&walk->iterator_time, 32, commit_time_cmp)) < GIT_SUCCESS) - return error; - - walk->get_next = &revwalk_next_timesort; - walk->enqueue = &revwalk_enqueue_timesort; - } else { - walk->get_next = &revwalk_next_unsorted; - walk->enqueue = &revwalk_enqueue_unsorted; - } - - for (i = 0; i < walk->pending.length; ++i) { - commit_object *commit = walk->pending.contents[i]; - if ((error = process_commit(walk, commit)) < GIT_SUCCESS) { - return error; - } - } + commit_object *next; if (walk->sorting & GIT_SORT_TOPOLOGICAL) { - commit_object *next; unsigned short i; - int error; while ((error = walk->get_next(&next, walk)) == GIT_SUCCESS) { for (i = 0; i < next->out_degree; ++i) { @@ -514,8 +427,6 @@ static int prepare_walk(git_revwalk *walk) } if (walk->sorting & GIT_SORT_REVERSE) { - commit_object *next; - int error; while ((error = walk->get_next(&next, walk)) == GIT_SUCCESS) commit_list_insert(next, &walk->iterator_reverse); @@ -531,6 +442,83 @@ static int prepare_walk(git_revwalk *walk) } + + + +int git_revwalk_new(git_revwalk **revwalk_out, git_repository *repo) +{ + git_revwalk *walk; + + walk = git__malloc(sizeof(git_revwalk)); + if (walk == NULL) + return GIT_ENOMEM; + + memset(walk, 0x0, sizeof(git_revwalk)); + + walk->commits = git_hashtable_alloc(64, + object_table_hash, + (git_hash_keyeq_ptr)git_oid_cmp); + + if (walk->commits == NULL) { + free(walk); + return GIT_ENOMEM; + } + + git_pqueue_init(&walk->iterator_time, 8, commit_time_cmp); + git_vector_init(&walk->memory_alloc, 8, NULL); + alloc_chunk(walk); + + walk->get_next = &revwalk_next_unsorted; + walk->enqueue = &revwalk_enqueue_unsorted; + + walk->repo = repo; + + *revwalk_out = walk; + return GIT_SUCCESS; +} + +void git_revwalk_free(git_revwalk *walk) +{ + unsigned int i; + + if (walk == NULL) + return; + + git_revwalk_reset(walk); + git_hashtable_free(walk->commits); + git_pqueue_free(&walk->iterator_time); + + for (i = 0; i < walk->memory_alloc.length; ++i) + free(git_vector_get(&walk->memory_alloc, i)); + + git_vector_free(&walk->memory_alloc); + free(walk); +} + +git_repository *git_revwalk_repository(git_revwalk *walk) +{ + assert(walk); + return walk->repo; +} + +void git_revwalk_sorting(git_revwalk *walk, unsigned int sort_mode) +{ + assert(walk); + + if (walk->walking) + git_revwalk_reset(walk); + + walk->sorting = sort_mode; + + if (walk->sorting & GIT_SORT_TIME) { + walk->get_next = &revwalk_next_timesort; + walk->enqueue = &revwalk_enqueue_timesort; + } else { + walk->get_next = &revwalk_next_unsorted; + walk->enqueue = &revwalk_enqueue_unsorted; + } +} + int git_revwalk_next(git_oid *oid, git_revwalk *walk) { int error; @@ -544,8 +532,11 @@ int git_revwalk_next(git_oid *oid, git_revwalk *walk) } error = walk->get_next(&next, walk); - if (error < GIT_SUCCESS) + if (error < GIT_SUCCESS) { + if (error == GIT_EREVWALKOVER) + git_revwalk_reset(walk); return error; + } git_oid_cpy(oid, &next->oid); return GIT_SUCCESS; @@ -564,7 +555,7 @@ void git_revwalk_reset(git_revwalk *walk) commit->topo_delay = 0; ); - git_pqueue_free(&walk->iterator_time); + git_pqueue_clear(&walk->iterator_time); commit_list_free(&walk->iterator_topo); commit_list_free(&walk->iterator_rand); commit_list_free(&walk->iterator_reverse); diff --git a/tests/t05-revwalk.c b/tests/t05-revwalk.c index bdec09e83..0460d4c08 100644 --- a/tests/t05-revwalk.c +++ b/tests/t05-revwalk.c @@ -84,7 +84,7 @@ static int get_commit_index(git_oid *raw_oid) return -1; } -static int test_walk(git_revwalk *walk, +static int test_walk(git_revwalk *walk, const git_oid *root, int flags, const int possible_results[][6], int results_count) { git_oid oid; @@ -92,8 +92,8 @@ static int test_walk(git_revwalk *walk, int i; int result_array[commit_count]; - git_revwalk_reset(walk); git_revwalk_sorting(walk, flags); + git_revwalk_push(walk, root); for (i = 0; i < commit_count; ++i) result_array[i] = -1; @@ -124,19 +124,14 @@ BEGIN_TEST(walk0, "do a simple walk on a repo with different sorting modes") git_revwalk *walk; must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); - must_pass(git_revwalk_new(&walk, repo)); git_oid_mkstr(&id, commit_head); - git_revwalk_push(walk, &id); - must_pass(test_walk(walk, GIT_SORT_TIME, commit_sorting_time, 1)); - - must_pass(test_walk(walk, GIT_SORT_TOPOLOGICAL, commit_sorting_topo, 2)); - - must_pass(test_walk(walk, GIT_SORT_TIME | GIT_SORT_REVERSE, commit_sorting_time_reverse, 1)); - - must_pass(test_walk(walk, GIT_SORT_TOPOLOGICAL | GIT_SORT_REVERSE, commit_sorting_topo_reverse, 2)); + must_pass(test_walk(walk, &id, GIT_SORT_TIME, commit_sorting_time, 1)); + must_pass(test_walk(walk, &id, GIT_SORT_TOPOLOGICAL, commit_sorting_topo, 2)); + must_pass(test_walk(walk, &id, GIT_SORT_TIME | GIT_SORT_REVERSE, commit_sorting_time_reverse, 1)); + must_pass(test_walk(walk, &id, GIT_SORT_TOPOLOGICAL | GIT_SORT_REVERSE, commit_sorting_topo_reverse, 2)); git_revwalk_free(walk); git_repository_free(repo); From 9f81a37aa416446726a00996dce01b02372a43b1 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 16 Mar 2011 23:02:31 +0200 Subject: [PATCH 26/81] Define NO_GZIP in zconf.h instead of at compile time --- CMakeLists.txt | 3 --- deps/zlib/zconf.h | 1 + wscript | 3 --- 3 files changed, 1 insertion(+), 6 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index e936bcf1d..1bd243099 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -75,9 +75,6 @@ FILE(GLOB SRC_SHA1 src/block-sha1/*.c) FILE(GLOB SRC_PLAT src/unix/*.c) FILE(GLOB SRC_H include/git2/*.h) -# Do not build Zlib with GZIP support -ADD_DEFINITIONS(-DNO_GZIP) - # On Windows use specific platform sources IF (WIN32 AND NOT CYGWIN) ADD_DEFINITIONS(-DWIN32 -D_DEBUG -D_LIB) diff --git a/deps/zlib/zconf.h b/deps/zlib/zconf.h index ea765a54c..494992aba 100644 --- a/deps/zlib/zconf.h +++ b/deps/zlib/zconf.h @@ -10,6 +10,7 @@ #include "../../src/common.h" +#define NO_GZIP #define STDC /* Jeez, don't complain about non-prototype diff --git a/wscript b/wscript index 6d5579fef..4e8294c9f 100644 --- a/wscript +++ b/wscript @@ -61,9 +61,6 @@ def configure(conf): conf.env.PLATFORM = 'unix' conf.check_cc(lib='pthread', uselib_store='pthread') - # Do not build ZLib with GZIP support - conf.env.DEFINES += ['NO_GZIP'] - # check for sqlite3 if conf.options.use_sqlite and conf.check_cc( lib='sqlite3', uselib_store='sqlite3', install_path=None, mandatory=False): From 7341bf87b111dfa0cf12761389e0e7a118363f42 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 16 Mar 2011 23:13:28 +0200 Subject: [PATCH 27/81] Refs are now always in-sync on disk --- src/refs.c | 185 +++++++++++++++++++++++++++++++++++++---------------- src/refs.h | 2 + 2 files changed, 131 insertions(+), 56 deletions(-) diff --git a/src/refs.c b/src/refs.c index ea39608df..40b80ec9b 100644 --- a/src/refs.c +++ b/src/refs.c @@ -59,16 +59,16 @@ static uint32_t reftable_hash(const void *key, int hash_id) static void reference_free(git_reference *reference); static int reference_create(git_reference **ref_out, git_repository *repo, const char *name, git_rtype type); +static int reference_read(gitfo_buf *file_content, time_t *mtime, const char *repo_path, const char *ref_name); /* loose refs */ static int loose_parse_symbolic(git_reference *ref, gitfo_buf *file_content); static int loose_parse_oid(git_reference *ref, gitfo_buf *file_content); -static int loose_read(gitfo_buf *file_content, const char *name, const char *repo_path); -static int loose_lookup( git_reference **ref_out, git_repository *repo, const char *name, int skip_symbolic); +static int loose_lookup(git_reference **ref_out, git_repository *repo, const char *name, int skip_symbolic); static int loose_write(git_reference *ref); +static int loose_update(git_reference *ref); /* packed refs */ -static int packed_readpack(gitfo_buf *packfile, const char *repo_path); static int packed_parse_peel(reference_oid *tag_ref, const char **buffer_out, const char *buffer_end); static int packed_parse_oid(reference_oid **ref_out, git_repository *repo, const char **buffer_out, const char *buffer_end); static int packed_load(git_repository *repo); @@ -146,12 +146,73 @@ cleanup: return error; } +static int reference_read(gitfo_buf *file_content, time_t *mtime, const char *repo_path, const char *ref_name) +{ + struct stat st; + char path[GIT_PATH_MAX]; + + /* Determine the full path of the file */ + git__joinpath(path, repo_path, ref_name); + + if (gitfo_stat(path, &st) < 0) + return GIT_ENOTFOUND; + + if (S_ISDIR(st.st_mode)) + return GIT_EOBJCORRUPTED; + + if (mtime) + *mtime = st.st_mtime; + + if (file_content) + return gitfo_read_file(file_content, path); + + return GIT_SUCCESS; +} + /***************************************** * Internal methods - Loose references *****************************************/ +static int loose_update(git_reference *ref) +{ + int error; + time_t ref_time; + gitfo_buf ref_file = GITFO_BUF_INIT; + + if (ref->type & GIT_REF_PACKED) + return packed_load(ref->owner); + + error = reference_read(NULL, &ref_time, ref->owner->path_repository, ref->name); + if (error < GIT_SUCCESS) + goto cleanup; + + if (ref_time == ref->mtime) + return GIT_SUCCESS; + + error = reference_read(&ref_file, &ref->mtime, ref->owner->path_repository, ref->name); + if (error < GIT_SUCCESS) + goto cleanup; + + if (ref->type == GIT_REF_SYMBOLIC) + error = loose_parse_symbolic(ref, &ref_file); + else if (ref->type == GIT_REF_OID) + error = loose_parse_oid(ref, &ref_file); + else + error = GIT_EINVALIDREFSTATE; + + gitfo_free_buf(&ref_file); + +cleanup: + if (error != GIT_SUCCESS) { + reference_free(ref); + git_hashtable_remove(ref->owner->references.loose_cache, ref->name); + } + + return error; +} + static int loose_parse_symbolic(git_reference *ref, gitfo_buf *file_content) { const unsigned int header_len = strlen(GIT_SYMREF); @@ -172,6 +233,7 @@ static int loose_parse_symbolic(git_reference *ref, gitfo_buf *file_content) refname_start += header_len; + free(ref_sym->target); ref_sym->target = git__strdup(refname_start); if (ref_sym->target == NULL) return GIT_ENOMEM; @@ -213,27 +275,6 @@ static int loose_parse_oid(git_reference *ref, gitfo_buf *file_content) return GIT_SUCCESS; } -static int loose_read(gitfo_buf *file_content, const char *name, const char *repo_path) -{ - int error = GIT_SUCCESS; - char ref_path[GIT_PATH_MAX]; - - /* Determine the full path of the ref */ - git__joinpath(ref_path, repo_path, name); - - /* Does it even exist ? */ - if (gitfo_exists(ref_path) < GIT_SUCCESS) - return GIT_ENOTFOUND; - - /* A ref can not be a directory */ - if (!gitfo_isdir(ref_path)) - return GIT_ENOTFOUND; - - if (file_content != NULL) - error = gitfo_read_file(file_content, ref_path); - - return error; -} static git_rtype loose_guess_rtype(const char *full_path) { @@ -262,10 +303,11 @@ static int loose_lookup( int error = GIT_SUCCESS; gitfo_buf ref_file = GITFO_BUF_INIT; git_reference *ref = NULL; + time_t ref_time; *ref_out = NULL; - error = loose_read(&ref_file, name, repo->path_repository); + error = reference_read(&ref_file, &ref_time, repo->path_repository, name); if (error < GIT_SUCCESS) goto cleanup; @@ -289,6 +331,7 @@ static int loose_lookup( if (error < GIT_SUCCESS) goto cleanup; + ref->mtime = ref_time; *ref_out = ref; return GIT_SUCCESS; @@ -304,6 +347,7 @@ static int loose_write(git_reference *ref) char ref_path[GIT_PATH_MAX]; int error, contents_size; char *ref_contents = NULL; + struct stat st; assert((ref->type & GIT_REF_PACKED) == 0); @@ -349,6 +393,9 @@ static int loose_write(git_reference *ref) error = git_filebuf_commit(&file); + if (gitfo_stat(ref_path, &st) == GIT_SUCCESS) + ref->mtime = st.st_mtime; + free(ref_contents); return error; @@ -366,19 +413,6 @@ unlock: /***************************************** * Internal methods - Packed references *****************************************/ -static int packed_readpack(gitfo_buf *packfile, const char *repo_path) -{ - char ref_path[GIT_PATH_MAX]; - - /* Determine the full path of the file */ - git__joinpath(ref_path, repo_path, GIT_PACKEDREFS_FILE); - - /* Does it even exist ? */ - if (gitfo_exists(ref_path) < GIT_SUCCESS) - return GIT_ENOTFOUND; - - return gitfo_read_file(packfile, ref_path); -} static int packed_parse_peel( reference_oid *tag_ref, @@ -483,19 +517,40 @@ static int packed_load(git_repository *repo) git_refcache *ref_cache = &repo->references; /* already loaded */ - if (repo->references.packfile != NULL) - return GIT_SUCCESS; + if (repo->references.packfile != NULL) { + time_t packed_time; - repo->references.packfile = git_hashtable_alloc( - default_table_size, - reftable_hash, - (git_hash_keyeq_ptr)strcmp); + /* check if we can read the time of the index; + * if we can read it and it matches the time of the + * index we had previously loaded, we don't need to do + * anything else. + * + * if we cannot load the time (e.g. the packfile + * has disappeared) or the time is different, we + * have to reload the packfile */ - if (repo->references.packfile == NULL) - return GIT_ENOMEM; + if (!reference_read(NULL, &packed_time, repo->path_repository, GIT_PACKEDREFS_FILE) && + packed_time == ref_cache->packfile_time) + return GIT_SUCCESS; - /* read the packfile from disk */ - error = packed_readpack(&packfile, repo->path_repository); + git_hashtable_clear(repo->references.packfile); + } else { + ref_cache->packfile = git_hashtable_alloc( + default_table_size, + reftable_hash, + (git_hash_keyeq_ptr)strcmp); + + if (ref_cache->packfile == NULL) + return GIT_ENOMEM; + } + + /* read the packfile from disk; + * store its modification time to check for future reloads */ + error = reference_read( + &packfile, + &ref_cache->packfile_time, + repo->path_repository, + GIT_PACKEDREFS_FILE); /* there is no packfile on disk; that's ok */ if (error == GIT_ENOTFOUND) @@ -536,7 +591,12 @@ static int packed_load(git_repository *repo) } } + gitfo_free_buf(&packfile); + return GIT_SUCCESS; + cleanup: + git_hashtable_free(ref_cache->packfile); + ref_cache->packfile = NULL; gitfo_free_buf(&packfile); return error; } @@ -834,8 +894,14 @@ cleanup: /* when and only when the packfile has been properly written, * we can go ahead and remove the loose refs */ - if (error == GIT_SUCCESS) + if (error == GIT_SUCCESS) { + struct stat st; + error = packed_remove_loose(repo, &packing_list); + + if (gitfo_stat(pack_file_path, &st) == GIT_SUCCESS) + repo->references.packfile_time = st.st_mtime; + } } else git_filebuf_cleanup(&pack_file); @@ -870,7 +936,7 @@ int git_reference_lookup(git_reference **ref_out, git_repository *repo, const ch /* First, check has been previously loaded and cached */ *ref_out = git_hashtable_lookup(repo->references.loose_cache, normalized_name); if (*ref_out != NULL) - return GIT_SUCCESS; + return loose_update(*ref_out); /* Then check if there is a loose file for that reference.*/ error = loose_lookup(ref_out, repo, normalized_name, 0); @@ -888,12 +954,10 @@ int git_reference_lookup(git_reference **ref_out, git_repository *repo, const ch * If we cannot find a loose reference, we look into the packfile * Load the packfile first if it hasn't been loaded */ - if (!repo->references.packfile) { - /* load all the packed references */ - error = packed_load(repo); - if (error < GIT_SUCCESS) - return error; - } + /* load all the packed references */ + error = packed_load(repo); + if (error < GIT_SUCCESS) + return error; /* Look up on the packfile */ *ref_out = git_hashtable_lookup(repo->references.packfile, normalized_name); @@ -1000,6 +1064,9 @@ const git_oid *git_reference_oid(git_reference *ref) if ((ref->type & GIT_REF_OID) == 0) return NULL; + if (loose_update(ref) < GIT_SUCCESS) + return NULL; + return &((reference_oid *)ref)->oid; } @@ -1010,6 +1077,9 @@ const char *git_reference_target(git_reference *ref) if ((ref->type & GIT_REF_SYMBOLIC) == 0) return NULL; + if (loose_update(ref) < GIT_SUCCESS) + return NULL; + return ((reference_symbolic *)ref)->target; } @@ -1293,6 +1363,9 @@ int git_reference_resolve(git_reference **resolved_ref, git_reference *ref) assert(resolved_ref && ref); *resolved_ref = NULL; + + if ((error = loose_update(ref)) < GIT_SUCCESS) + return error; repo = ref->owner; diff --git a/src/refs.h b/src/refs.h index a542ac0f2..bebb1b97d 100644 --- a/src/refs.h +++ b/src/refs.h @@ -23,11 +23,13 @@ struct git_reference { git_repository *owner; char *name; unsigned int type; + time_t mtime; }; typedef struct { git_hashtable *packfile; git_hashtable *loose_cache; + time_t packfile_time; } git_refcache; From b5c5f0f8086ee4e9bccf0703386fd5219ac380c2 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 16 Mar 2011 23:59:09 +0200 Subject: [PATCH 28/81] Fix headers for the new Revision Walker The "oid.h" header is now included instead of "object.h". The old "revwalk.h" header has been removed; it was empty. --- include/git2/revwalk.h | 2 +- src/commit.c | 1 - src/revwalk.c | 3 ++- src/revwalk.h | 11 ----------- src/tag.c | 1 - src/tree.c | 1 - tests/t05-revwalk.c | 2 -- 7 files changed, 3 insertions(+), 18 deletions(-) delete mode 100644 src/revwalk.h diff --git a/include/git2/revwalk.h b/include/git2/revwalk.h index 4b5ea8e06..f3e0152d4 100644 --- a/include/git2/revwalk.h +++ b/include/git2/revwalk.h @@ -27,7 +27,7 @@ #include "common.h" #include "types.h" -#include "object.h" +#include "oid.h" /** * @file git2/revwalk.h diff --git a/src/commit.c b/src/commit.c index 1c5cddf7a..8da170cad 100644 --- a/src/commit.c +++ b/src/commit.c @@ -30,7 +30,6 @@ #include "common.h" #include "commit.h" -#include "revwalk.h" #include "signature.h" #define COMMIT_BASIC_PARSE 0x0 diff --git a/src/revwalk.c b/src/revwalk.c index a2fc092d5..3b7ad34e6 100644 --- a/src/revwalk.c +++ b/src/revwalk.c @@ -25,10 +25,11 @@ #include "common.h" #include "commit.h" -#include "revwalk.h" #include "hashtable.h" #include "pqueue.h" +#include "git2/revwalk.h" + typedef struct commit_object { git_oid oid; uint32_t time; diff --git a/src/revwalk.h b/src/revwalk.h deleted file mode 100644 index 2970d773c..000000000 --- a/src/revwalk.h +++ /dev/null @@ -1,11 +0,0 @@ -#ifndef INCLUDE_revwalk_h__ -#define INCLUDE_revwalk_h__ - -#include "git2/common.h" -#include "git2/revwalk.h" - -#include "commit.h" -#include "repository.h" -#include "hashtable.h" - -#endif /* INCLUDE_revwalk_h__ */ diff --git a/src/tag.c b/src/tag.c index 1379425a1..0489e7e4a 100644 --- a/src/tag.c +++ b/src/tag.c @@ -27,7 +27,6 @@ #include "commit.h" #include "tag.h" #include "signature.h" -#include "revwalk.h" #include "git2/object.h" #include "git2/repository.h" #include "git2/signature.h" diff --git a/src/tree.c b/src/tree.c index 702cccbce..307d41b0d 100644 --- a/src/tree.c +++ b/src/tree.c @@ -25,7 +25,6 @@ #include "common.h" #include "commit.h" -#include "revwalk.h" #include "tree.h" #include "git2/repository.h" #include "git2/object.h" diff --git a/tests/t05-revwalk.c b/tests/t05-revwalk.c index 0460d4c08..cfcf01066 100644 --- a/tests/t05-revwalk.c +++ b/tests/t05-revwalk.c @@ -25,8 +25,6 @@ #include "test_lib.h" #include "test_helpers.h" -#include "revwalk.h" - /* $ git log --oneline --graph --decorate * a4a7dce (HEAD, br2) Merge branch 'master' into br2 From bb3de0c472b2d5d6b8091c190bee3db79c4b5e27 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 16 Mar 2011 21:35:51 +0200 Subject: [PATCH 29/81] Thread safe cache --- include/git2/thread-utils.h | 2 - src/cache.c | 175 ++++++++++++++++++++++++++++++++++++ src/cache.h | 38 ++++++++ src/common.h | 9 +- src/thread-utils.h | 82 +++++++++++++++-- src/win32/pthread.c | 64 ++++--------- src/win32/pthread.h | 9 +- tests/t11-sqlite.c | 20 +++-- tests/t13-threads.c | 48 ++++++++++ tests/test_main.c | 2 + wscript | 10 ++- 11 files changed, 382 insertions(+), 77 deletions(-) create mode 100644 src/cache.c create mode 100644 src/cache.h create mode 100644 tests/t13-threads.c diff --git a/include/git2/thread-utils.h b/include/git2/thread-utils.h index ecb09bb53..fb8644b93 100644 --- a/include/git2/thread-utils.h +++ b/include/git2/thread-utils.h @@ -67,6 +67,4 @@ # define GIT_TLS #endif -#define GIT_THREADS 1 - #endif /* INCLUDE_git_thread_utils_h__ */ diff --git a/src/cache.c b/src/cache.c new file mode 100644 index 000000000..5a9b8415c --- /dev/null +++ b/src/cache.c @@ -0,0 +1,175 @@ +/* + * This file is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License, version 2, + * as published by the Free Software Foundation. + * + * In addition to the permissions in the GNU General Public License, + * the authors give you unlimited permission to link the compiled + * version of this file into combinations with other programs, + * and to distribute those combinations without any restriction + * coming from the use of this file. (The General Public License + * restrictions do apply in other respects; for example, they cover + * modification of the file, and distribution when not linked into + * a combined executable.) + * + * This file is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; see the file COPYING. If not, write to + * the Free Software Foundation, 51 Franklin Street, Fifth Floor, + * Boston, MA 02110-1301, USA. + */ + +#include "common.h" +#include "repository.h" +#include "commit.h" +#include "thread-utils.h" +#include "cache.h" + +#define GIT_CACHE_OPENADR 3 + + +GIT_INLINE(int) cached_obj_compare(git_cached_obj *obj, const git_oid *oid) +{ + return git_oid_cmp(&obj->oid, oid); +} + +GIT_INLINE(void) cached_obj_incref(git_cached_obj *obj) +{ + git_atomic_inc(&obj->refcount); +} + +GIT_INLINE(void) cached_obj_decref(git_cached_obj *obj, git_cached_obj_freeptr free_obj) +{ + if (git_atomic_dec(&obj->refcount) == 0) + free_obj(obj); +} + + +void git_cache_init(git_cache *cache, size_t size, git_cached_obj_freeptr free_ptr) +{ + size_t i; + + if (size < 8) + size = 8; + + /* round up size to closest power of 2 */ + size--; + size |= size >> 1; + size |= size >> 2; + size |= size >> 4; + size |= size >> 8; + size |= size >> 16; + + cache->size_mask = size; + cache->lru_count = 0; + cache->free_obj = free_ptr; + + cache->nodes = git__malloc((size + 1) * sizeof(cache_node)); + + for (i = 0; i < (size + 1); ++i) { + git_mutex_init(&cache->nodes[i].lock); + cache->nodes[i].ptr = NULL; + cache->nodes[i].lru = 0; + } +} + +void git_cache_free(git_cache *cache) +{ + size_t i; + + for (i = 0; i < (cache->size_mask + 1); ++i) { + cached_obj_decref(cache->nodes[i].ptr, cache->free_obj); + git_mutex_free(&cache->nodes[i].lock); + } + + free(cache->nodes); +} + +void *git_cache_get(git_cache *cache, const git_oid *oid) +{ + const uint32_t *hash; + size_t i, pos, found = 0; + cache_node *node; + + hash = (const uint32_t *)oid->id; + + for (i = 0; !found && i < GIT_CACHE_OPENADR; ++i) { + pos = hash[i] & cache->size_mask; + node = &cache->nodes[pos]; + + git_mutex_lock(&node->lock); + { + if (cached_obj_compare(node->ptr, oid) == 0) { + cached_obj_incref(node->ptr); + node->lru = ++cache->lru_count; + found = 1; + } + } + git_mutex_unlock(&node->lock); + } + + + return found ? node->ptr : NULL; +} + +void *git_cache_try_store(git_cache *cache, void *entry) +{ + cache_node *nodes[GIT_CACHE_OPENADR], *lru_node; + const uint32_t *hash; + const git_oid *oid; + size_t i, stored = 0; + + oid = &((git_cached_obj*)entry)->oid; + hash = (const uint32_t *)oid->id; + + /* increase the refcount on this object, because + * the cache now owns it */ + cached_obj_incref(entry); + + for (i = 0; i < GIT_CACHE_OPENADR; ++i) { + size_t pos = hash[i] & cache->size_mask; + + nodes[i] = &cache->nodes[pos]; + git_mutex_lock(&nodes[i]->lock); + } + + lru_node = nodes[0]; + + for (i = 0; !stored && entry && i < GIT_CACHE_OPENADR; ++i) { + + if (nodes[i]->ptr == NULL) { + nodes[i]->ptr = entry; + nodes[i]->lru = ++cache->lru_count; + stored = 1; + } else if (cached_obj_compare(nodes[i]->ptr, oid) == 0) { + cached_obj_decref(entry, cache->free_obj); + entry = nodes[i]->ptr; + stored = 1; + } + + if (nodes[i]->lru < lru_node->lru) + lru_node = nodes[i]; + } + + if (!stored) { + void *old_entry = lru_node->ptr; + assert(old_entry); + + cached_obj_decref(old_entry, cache->free_obj); + lru_node->ptr = entry; + lru_node->lru = ++cache->lru_count; + } + + /* increase the refcount again, because we are + * returning it to the user */ + cached_obj_incref(entry); + + for (i = 0; i < GIT_CACHE_OPENADR; ++i) + git_mutex_unlock(&nodes[i]->lock); + + return entry; +} diff --git a/src/cache.h b/src/cache.h new file mode 100644 index 000000000..9f525e68c --- /dev/null +++ b/src/cache.h @@ -0,0 +1,38 @@ +#ifndef INCLUDE_cache_h__ +#define INCLUDE_cache_h__ + +#include "git2/common.h" +#include "git2/oid.h" +#include "git2/odb.h" + +#include "thread-utils.h" + +typedef void (*git_cached_obj_freeptr)(void *); + +typedef struct { + git_oid oid; + git_atomic refcount; +} git_cached_obj; + +typedef struct { + git_cached_obj *ptr; + git_mutex lock; + unsigned int lru; +} cache_node; + +typedef struct { + cache_node *nodes; + + unsigned int lru_count; + size_t size_mask; + git_cached_obj_freeptr free_obj; +} git_cache; + + +void git_cache_init(git_cache *cache, size_t size, git_cached_obj_freeptr free_ptr); +void git_cache_free(git_cache *cache); + +void *git_cache_try_store(git_cache *cache, void *entry); +void *git_cache_get(git_cache *cache, const git_oid *oid); + +#endif diff --git a/src/common.h b/src/common.h index 723085a2b..5ad878e26 100644 --- a/src/common.h +++ b/src/common.h @@ -31,18 +31,21 @@ # include # include "msvc-compat.h" # include "mingw-compat.h" -# include "win32/pthread.h" +# ifdef GIT_THREADS +# include "win32/pthread.h" +#endif # define snprintf _snprintf typedef SSIZE_T ssize_t; #else - # include # include -# include +# ifdef GIT_THREADS +# include +# endif #endif #include "git2/common.h" diff --git a/src/thread-utils.h b/src/thread-utils.h index e8372e731..1cf0e3407 100644 --- a/src/thread-utils.h +++ b/src/thread-utils.h @@ -2,6 +2,19 @@ #define INCLUDE_thread_utils_h__ + +/* Common operations even if threading has been disabled */ +typedef struct { + volatile int val; +} git_atomic; + +static inline void git_atomic_set(git_atomic *a, int val) +{ + a->val = val; +} + +#ifdef GIT_THREADS + #define git_thread pthread_t #define git_thread_create(thread, attr, start_routine, arg) pthread_create(thread, attr, start_routine, arg) #define git_thread_kill(thread) pthread_cancel(thread) @@ -15,13 +28,70 @@ #define git_mutex_unlock(a) pthread_mutex_unlock(a) #define git_mutex_free(a) pthread_mutex_destroy(a) +/* Pthreads condition vars -- disabled by now */ +#define git_cond unsigned int //pthread_cond_t +#define git_cond_init(c, a) (void)0 //pthread_cond_init(c, a) +#define git_cond_free(c) (void)0 //pthread_cond_destroy(c) +#define git_cond_wait(c, l) (void)0 //pthread_cond_wait(c, l) +#define git_cond_signal(c) (void)0 //pthread_cond_signal(c) +#define git_cond_broadcast(c) (void)0 //pthread_cond_broadcast(c) + +static inline int git_atomic_inc(git_atomic *a) +{ +#ifdef __GNUC__ + return __sync_add_and_fetch(&a->val, 1); +#elif defined(_MSC_VER) + return InterlockedIncrement(&a->val); +#else +# error "Unsupported architecture for atomic operations" +#endif +} + +static inline int git_atomic_dec(git_atomic *a) +{ +#ifdef __GNUC__ + return __sync_sub_and_fetch(&a->val, 1); +#elif defined(_MSC_VER) + return InterlockedDecrement(&a->val); +#else +# error "Unsupported architecture for atomic operations" +#endif +} + +#else + +#define git_thread unsigned int +#define git_thread_create(thread, attr, start_routine, arg) (void)0 +#define git_thread_kill(thread) (void)0 +#define git_thread_exit(status) (void)0 +#define git_thread_join(id, status) (void)0 + +/* Pthreads Mutex */ +#define git_mutex unsigned int +#define git_mutex_init(a) (void)0 +#define git_mutex_lock(a) (void)0 +#define git_mutex_unlock(a) (void)0 +#define git_mutex_free(a) (void)0 + /* Pthreads condition vars */ -#define git_cond pthread_cond_t -#define git_cond_init(c, a) pthread_cond_init(c, a) -#define git_cond_free(c) pthread_cond_destroy(c) -#define git_cond_wait(c, l) pthread_cond_wait(c, l) -#define git_cond_signal(c) pthread_cond_signal(c) -#define git_cond_broadcast(c) pthread_cond_broadcast(c) +#define git_cond unsigned int +#define git_cond_init(c, a) (void)0 +#define git_cond_free(c) (void)0 +#define git_cond_wait(c, l) (void)0 +#define git_cond_signal(c) (void)0 +#define git_cond_broadcast(c) (void)0 + +static inline int git_atomic_inc(git_atomic *a) +{ + return ++a->val; +} + +static inline int git_atomic_dec(git_atomic *a) +{ + return --a->val; +} + +#endif extern int git_online_cpus(void); diff --git a/src/win32/pthread.c b/src/win32/pthread.c index fffff81df..f47364a76 100644 --- a/src/win32/pthread.c +++ b/src/win32/pthread.c @@ -36,17 +36,27 @@ int pthread_create(pthread_t *GIT_RESTRICT thread, return *thread ? GIT_SUCCESS : GIT_EOSERR; } -int pthread_cond_signal(pthread_cond_t *cond) +int pthread_join(pthread_t thread, void **value_ptr) { - WakeConditionVariable(cond); + int ret; + ret = WaitForSingleObject(thread, INFINITE); + if (ret && value_ptr) + GetExitCodeThread(thread, (void*) value_ptr); + return -(!!ret); +} + +int pthread_mutex_init(pthread_mutex_t *GIT_RESTRICT mutex, + const pthread_mutexattr_t *GIT_RESTRICT GIT_UNUSED(mutexattr)) +{ + GIT_UNUSED_ARG(mutexattr); + InitializeCriticalSection(mutex); return 0; } -int pthread_cond_wait(pthread_cond_t *GIT_RESTRICT cond, - pthread_mutex_t *GIT_RESTRICT mutex) +int pthread_mutex_destroy(pthread_mutex_t *mutex) { int ret; - ret = SleepConditionVariableCS(cond, mutex, INFINITE); + ret = CloseHandle(mutex); return -(!ret); } @@ -62,50 +72,6 @@ int pthread_mutex_unlock(pthread_mutex_t *mutex) return 0; } -int pthread_join(pthread_t thread, void **value_ptr) -{ - int ret; - ret = WaitForSingleObject(thread, INFINITE); - if (ret && value_ptr) - GetExitCodeThread(thread, (void*) value_ptr); - return -(!!ret); -} - -int pthread_cond_broadcast(pthread_cond_t *cond) -{ - WakeAllConditionVariable(cond); - return 0; -} - -int pthread_mutex_destroy(pthread_mutex_t *mutex) -{ - int ret; - ret = CloseHandle(mutex); - return -(!ret); -} - -int pthread_cond_destroy(pthread_cond_t *GIT_UNUSED(cond)) -{ - GIT_UNUSED_ARG(cond); - return 0; -} - -int pthread_cond_init(pthread_cond_t *GIT_RESTRICT cond, - const pthread_condattr_t *GIT_RESTRICT GIT_UNUSED(condattr)) -{ - GIT_UNUSED_ARG(condattr); - InitializeConditionVariable(cond); - return 0; -} - -int pthread_mutex_init(pthread_mutex_t *GIT_RESTRICT mutex, - const pthread_mutexattr_t *GIT_RESTRICT GIT_UNUSED(mutexattr)) -{ - GIT_UNUSED_ARG(mutexattr); - InitializeCriticalSection(mutex); - return 0; -} - int pthread_num_processors_np(void) { DWORD_PTR p, s; diff --git a/src/win32/pthread.h b/src/win32/pthread.h index ff694e303..10949f1eb 100644 --- a/src/win32/pthread.h +++ b/src/win32/pthread.h @@ -40,7 +40,6 @@ typedef int pthread_mutexattr_t; typedef int pthread_condattr_t; typedef int pthread_attr_t; typedef CRITICAL_SECTION pthread_mutex_t; -typedef CONDITION_VARIABLE pthread_cond_t; typedef HANDLE pthread_t; #define PTHREAD_MUTEX_INITIALIZER {(void*)-1}; @@ -56,12 +55,6 @@ int pthread_mutex_destroy(pthread_mutex_t *); int pthread_mutex_lock(pthread_mutex_t *); int pthread_mutex_unlock(pthread_mutex_t *); -int pthread_cond_init(pthread_cond_t *GIT_RESTRICT, const pthread_condattr_t *GIT_RESTRICT); -int pthread_cond_destroy(pthread_cond_t *); -int pthread_cond_broadcast(pthread_cond_t *); -int pthread_cond_signal(pthread_cond_t *); -int pthread_cond_wait(pthread_cond_t *GIT_RESTRICT, pthread_mutex_t *GIT_RESTRICT); - int pthread_num_processors_np(void); -#endif \ No newline at end of file +#endif diff --git a/tests/t11-sqlite.c b/tests/t11-sqlite.c index 9e9d1b786..ba9065e76 100644 --- a/tests/t11-sqlite.c +++ b/tests/t11-sqlite.c @@ -23,11 +23,14 @@ * Boston, MA 02110-1301, USA. */ #include "test_lib.h" -#include "t03-data.h" + +#ifdef GIT2_SQLITE_BACKEND +#include "t03-data.h" #include "fileops.h" #include "git2/odb_backend.h" + static int cmp_objects(git_rawobj *o1, git_rawobj *o2) { if (o1->type != o2->type) @@ -41,7 +44,6 @@ static int cmp_objects(git_rawobj *o1, git_rawobj *o2) static git_odb *open_sqlite_odb(void) { -#ifdef GIT2_SQLITE_BACKEND git_odb *odb; git_odb_backend *sqlite; @@ -55,9 +57,6 @@ static git_odb *open_sqlite_odb(void) return NULL; return odb; -#else - return NULL; -#endif } #define TEST_WRITE(PTR) {\ @@ -105,7 +104,6 @@ END_TEST BEGIN_SUITE(sqlite) -#ifdef GIT2_SQLITE_BACKEND ADD_TEST(sqlite0); ADD_TEST(sqlite1); ADD_TEST(sqlite2); @@ -113,5 +111,13 @@ BEGIN_SUITE(sqlite) ADD_TEST(sqlite4); ADD_TEST(sqlite5); ADD_TEST(sqlite6); -#endif END_SUITE + +#else /* no sqlite builtin */ +BEGIN_SUITE(sqlite) + /* empty */ +END_SUITE +#endif + + + diff --git a/tests/t13-threads.c b/tests/t13-threads.c new file mode 100644 index 000000000..62ca73ac2 --- /dev/null +++ b/tests/t13-threads.c @@ -0,0 +1,48 @@ +/* + * This file is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License, version 2, + * as published by the Free Software Foundation. + * + * In addition to the permissions in the GNU General Public License, + * the authors give you unlimited permission to link the compiled + * version of this file into combinations with other programs, + * and to distribute those combinations without any restriction + * coming from the use of this file. (The General Public License + * restrictions do apply in other respects; for example, they cover + * modification of the file, and distribution when not linked into + * a combined executable.) + * + * This file is distributed in the hope that it will be useful, but + * WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; see the file COPYING. If not, write to + * the Free Software Foundation, 51 Franklin Street, Fifth Floor, + * Boston, MA 02110-1301, USA. + */ +#include "test_lib.h" +#include "test_helpers.h" +#include "cache.h" + + +typedef struct { + git_cached_obj cached; + unsigned int __dummy; +} ttest_obj; + +void *cache0_thread(void *data) +{ + git_cache *cache = (git_cache*)data; + unsigned int num; + +} + +BEGIN_TEST(cache0, "run several threads polling the cache at the same time") + +END_TEST + +BEGIN_SUITE(threads) + ADD_TEST(cache0); +END_SUITE diff --git a/tests/test_main.c b/tests/test_main.c index 9308b8d45..f2a623a48 100644 --- a/tests/test_main.c +++ b/tests/test_main.c @@ -42,6 +42,7 @@ DECLARE_SUITE(tree); DECLARE_SUITE(refs); DECLARE_SUITE(sqlite); DECLARE_SUITE(repository); +DECLARE_SUITE(threads); static libgit2_suite suite_methods[]= { SUITE_NAME(core), @@ -57,6 +58,7 @@ static libgit2_suite suite_methods[]= { SUITE_NAME(refs), SUITE_NAME(sqlite), SUITE_NAME(repository), + SUITE_NAME(threads), }; #define GIT_SUITE_COUNT (ARRAY_SIZE(suite_methods)) diff --git a/wscript b/wscript index 4e8294c9f..b990e148a 100644 --- a/wscript +++ b/wscript @@ -29,8 +29,10 @@ PPC optimized version (ppc) or the SHA1 functions from OpenSSL (openssl)") help='Force a specific MSVC++ version (7.1, 8.0, 9.0, 10.0), if more than one is installed') opt.add_option('--arch', action='store', default='x86', help='Select target architecture (ia64, x64, x86, x86_amd64, x86_ia64)') - opt.add_option('--without-sqlite', action='store_false', default=True, + opt.add_option('--with-sqlite', action='store_true', default=False, dest='use_sqlite', help='Disable sqlite support') + opt.add_option('--threadsafe', action='store_true', default=False, + help='Make libgit2 thread-safe (requires pthreads)') def configure(conf): @@ -59,7 +61,11 @@ def configure(conf): else: conf.env.PLATFORM = 'unix' - conf.check_cc(lib='pthread', uselib_store='pthread') + + if conf.options.threadsafe: + if conf.env.PLATFORM == 'unix': + conf.check_cc(lib='pthread', uselib_store='pthread') + conf.env.DEFINES += ['GIT_THREADS'] # check for sqlite3 if conf.options.use_sqlite and conf.check_cc( From 72a3fe42fb7208712bbe8f0981f4c6274c05e9c3 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Fri, 18 Mar 2011 19:38:49 +0200 Subject: [PATCH 30/81] I broke your bindings Hey. Apologies in advance -- I broke your bindings. This is a major commit that includes a long-overdue redesign of the whole object-database structure. This is expected to be the last major external API redesign of the library until the first non-alpha release. Please get your bindings up to date with these changes. They will be included in the next minor release. Sorry again! Major features include: - Real caching and refcounting on parsed objects - Real caching and refcounting on objects read from the ODB - Streaming writes & reads from the ODB - Single-method writes for all object types - The external API is now partially thread-safe The speed increases are significant in all aspects, specially when reading an object several times from the ODB (revwalking) and when writing big objects to the ODB. Here's a full changelog for the external API: blob.h ------ - Remove `git_blob_new` - Remove `git_blob_set_rawcontent` - Remove `git_blob_set_rawcontent_fromfile` - Rename `git_blob_writefile` -> `git_blob_create_fromfile` - Change `git_blob_create_fromfile`: The `path` argument is now relative to the repository's working dir - Add `git_blob_create_frombuffer` commit.h -------- - Remove `git_commit_new` - Remove `git_commit_add_parent` - Remove `git_commit_set_message` - Remove `git_commit_set_committer` - Remove `git_commit_set_author` - Remove `git_commit_set_tree` - Add `git_commit_create` - Add `git_commit_create_v` - Add `git_commit_create_o` - Add `git_commit_create_ov` tag.h ----- - Remove `git_tag_new` - Remove `git_tag_set_target` - Remove `git_tag_set_name` - Remove `git_tag_set_tagger` - Remove `git_tag_set_message` - Add `git_tag_create` - Add `git_tag_create_o` tree.h ------ - Change `git_tree_entry_2object`: New signature is `(git_object **object_out, git_repository *repo, git_tree_entry *entry)` - Remove `git_tree_new` - Remove `git_tree_add_entry` - Remove `git_tree_remove_entry_byindex` - Remove `git_tree_remove_entry_byname` - Remove `git_tree_clearentries` - Remove `git_tree_entry_set_id` - Remove `git_tree_entry_set_name` - Remove `git_tree_entry_set_attributes` object.h ------------ - Remove `git_object_new - Remove `git_object_write` - Change `git_object_close`: This method is now *mandatory*. Not closing an object causes a memory leak. odb.h ----- - Remove type `git_rawobj` - Remove `git_rawobj_close` - Rename `git_rawobj_hash` -> `git_odb_hash` - Change `git_odb_hash`: New signature is `(git_oid *id, const void *data, size_t len, git_otype type)` - Add type `git_odb_object` - Add `git_odb_object_close` - Change `git_odb_read`: New signature is `(git_odb_object **out, git_odb *db, const git_oid *id)` - Change `git_odb_read_header`: New signature is `(size_t *len_p, git_otype *type_p, git_odb *db, const git_oid *id)` - Remove `git_odb_write` - Add `git_odb_open_wstream` - Add `git_odb_open_rstream` odb_backend.h ------------- - Change type `git_odb_backend`: New internal signatures are as follows int (* read)(void **, size_t *, git_otype *, struct git_odb_backend *, const git_oid *) int (* read_header)(size_t *, git_otype *, struct git_odb_backend *, const git_oid *) int (* writestream)(struct git_odb_stream **, struct git_odb_backend *, size_t, git_otype) int (* readstream)( struct git_odb_stream **, struct git_odb_backend *, const git_oid *) - Add type `git_odb_stream` - Add enum `git_odb_streammode` Signed-off-by: Vicent Marti --- include/git2/blob.h | 67 +++------ include/git2/commit.h | 161 ++++++++++++++------ include/git2/common.h | 3 + include/git2/object.h | 59 +------- include/git2/odb.h | 133 +++++++++++------ include/git2/odb_backend.h | 40 ++++- include/git2/tag.h | 101 ++++++++----- include/git2/tree.h | 116 ++------------- include/git2/types.h | 7 +- src/backends/sqlite.c | 9 ++ src/blob.c | 135 ++++++++--------- src/blob.h | 6 +- src/cache.c | 46 ++---- src/cache.h | 21 +++ src/commit.c | 278 ++++++++++++++++++++--------------- src/commit.h | 4 +- src/delta-apply.h | 2 + src/errors.c | 3 +- src/filebuf.c | 190 ++++++++++++++++++------ src/filebuf.h | 22 ++- src/fileops.c | 90 +++++++++++- src/fileops.h | 2 + src/index.c | 2 +- src/object.c | 276 +++-------------------------------- src/odb.c | 143 +++++++++++------- src/odb.h | 18 ++- src/odb_loose.c | 292 +++++++++++++++++++------------------ src/odb_pack.c | 19 ++- src/oid.c | 14 +- src/repository.c | 72 +-------- src/repository.h | 28 +--- src/revwalk.c | 13 +- src/signature.c | 23 +-- src/signature.h | 2 +- src/tag.c | 144 +++++++++--------- src/tag.h | 4 +- src/thread-utils.h | 10 +- src/tree.c | 191 ++---------------------- src/tree.h | 7 +- src/util.h | 2 + tests/t01-rawobj.c | 37 +++-- tests/t02-objread.c | 71 ++++----- tests/t03-objwrite.c | 73 ++++++---- tests/t04-commit.c | 75 +++------- tests/t08-tag.c | 49 ++++++- tests/t09-tree.c | 84 +---------- tests/t11-sqlite.c | 4 +- tests/test_helpers.h | 2 + 48 files changed, 1468 insertions(+), 1682 deletions(-) diff --git a/include/git2/blob.h b/include/git2/blob.h index 2b7154fb5..3cd1467bf 100644 --- a/include/git2/blob.h +++ b/include/git2/blob.h @@ -41,8 +41,6 @@ GIT_BEGIN_DECL /** * Lookup a blob object from a repository. - * The generated blob object is owned by the revision - * repo and shall not be freed by the user. * * @param blob pointer to the looked up blob * @param repo the repo to use when locating the blob. @@ -54,50 +52,13 @@ GIT_INLINE(int) git_blob_lookup(git_blob **blob, git_repository *repo, const git return git_object_lookup((git_object **)blob, repo, id, GIT_OBJ_BLOB); } -/** - * Create a new in-memory git_blob. - * - * The blob object must be manually filled using - * the 'set_rawcontent' methods before it can - * be written back to disk. - * - * @param blob pointer to the new blob - * @param repo The repository where the object will reside - * @return 0 on success; error code otherwise - */ -GIT_INLINE(int) git_blob_new(git_blob **blob, git_repository *repo) -{ - return git_object_new((git_object **)blob, repo, GIT_OBJ_BLOB); -} - -/** - * Fill a blob with the contents inside - * the pointed file. - * - * @param blob pointer to the new blob - * @param filename name of the file to read - * @return 0 on success; error code otherwise - */ -GIT_EXTERN(int) git_blob_set_rawcontent_fromfile(git_blob *blob, const char *filename); - -/** - * Fill a blob with the contents inside - * the pointed buffer - * - * @param blob pointer to the blob - * @param buffer buffer with the contents for the blob - * @param len size of the buffer - * @return 0 on success; error code otherwise - */ -GIT_EXTERN(int) git_blob_set_rawcontent(git_blob *blob, const void *buffer, size_t len); - /** * Get a read-only buffer with the raw content of a blob. * * A pointer to the raw content of a blob is returned; * this pointer is owned internally by the object and shall * not be free'd. The pointer may be invalidated at a later - * time (e.g. when changing the contents of the blob). + * time. * * @param blob pointer to the blob * @return the pointer; NULL if the blob has no contents @@ -114,14 +75,28 @@ GIT_EXTERN(int) git_blob_rawsize(git_blob *blob); /** * Read a file from the working folder of a repository - * and write it to the Object Database as a loose blob, - * if such doesn't exist yet. + * and write it to the Object Database as a loose blob * - * @param written_id return the id of the written blob - * @param repo repository where the blob will be written - * @param path file from which the blob will be created + * @param oid return the id of the written blob + * @param repo repository where the blob will be written. + * this repository cannot be bare + * @param path file from which the blob will be created, + * relative to the repository's working dir + * @return 0 on success; error code otherwise */ -GIT_EXTERN(int) git_blob_writefile(git_oid *written_id, git_repository *repo, const char *path); +GIT_EXTERN(int) git_blob_create_fromfile(git_oid *oid, git_repository *repo, const char *path); + + +/** + * Write an in-memory buffer to the ODB as a blob + * + * @param oid return the oid of the written blob + * @param repo repository where to blob will be written + * @param buffer data to be written into the blob + * @param len length of the data + * @return 0 on success; error code otherwise + */ +GIT_EXTERN(int) git_blob_create_frombuffer(git_oid *oid, git_repository *repo, const void *buffer, size_t len); /** @} */ GIT_END_DECL diff --git a/include/git2/commit.h b/include/git2/commit.h index 1556e52b1..ba18a5b39 100644 --- a/include/git2/commit.h +++ b/include/git2/commit.h @@ -41,8 +41,6 @@ GIT_BEGIN_DECL /** * Lookup a commit object from a repository. - * The generated commit object is owned by the revision - * repo and shall not be freed by the user. * * @param commit pointer to the looked up commit * @param repo the repo to use when locating the commit. @@ -55,24 +53,9 @@ GIT_INLINE(int) git_commit_lookup(git_commit **commit, git_repository *repo, con return git_object_lookup((git_object **)commit, repo, id, GIT_OBJ_COMMIT); } -/** - * Create a new in-memory git_commit. - * - * The commit object must be manually filled using - * setter methods before it can be written to its - * repository. - * - * @param commit pointer to the new commit - * @param repo The repository where the object will reside - * @return 0 on success; error code otherwise - */ -GIT_INLINE(int) git_commit_new(git_commit **commit, git_repository *repo) -{ - return git_object_new((git_object **)commit, repo, GIT_OBJ_COMMIT); -} - /** * Get the id of a commit. + * * @param commit a previously loaded commit. * @return object identity for the commit. */ @@ -80,6 +63,7 @@ GIT_EXTERN(const git_oid *) git_commit_id(git_commit *commit); /** * Get the short (one line) message of a commit. + * * @param commit a previously loaded commit. * @return the short message of a commit */ @@ -87,6 +71,7 @@ GIT_EXTERN(const char *) git_commit_message_short(git_commit *commit); /** * Get the full message of a commit. + * * @param commit a previously loaded commit. * @return the message of a commit */ @@ -94,6 +79,7 @@ GIT_EXTERN(const char *) git_commit_message(git_commit *commit); /** * Get the commit time (i.e. committer time) of a commit. + * * @param commit a previously loaded commit. * @return the time of a commit */ @@ -101,6 +87,7 @@ GIT_EXTERN(time_t) git_commit_time(git_commit *commit); /** * Get the commit timezone offset (i.e. committer's preferred timezone) of a commit. + * * @param commit a previously loaded commit. * @return positive or negative timezone offset, in minutes from UTC */ @@ -108,6 +95,7 @@ GIT_EXTERN(int) git_commit_time_offset(git_commit *commit); /** * Get the committer of a commit. + * * @param commit a previously loaded commit. * @return the committer of a commit */ @@ -115,6 +103,7 @@ GIT_EXTERN(const git_signature *) git_commit_committer(git_commit *commit); /** * Get the author of a commit. + * * @param commit a previously loaded commit. * @return the author of a commit */ @@ -122,6 +111,7 @@ GIT_EXTERN(const git_signature *) git_commit_author(git_commit *commit); /** * Get the tree pointed to by a commit. + * * @param tree_out pointer where to store the tree object * @param commit a previously loaded commit. * @return 0 on success; error code otherwise @@ -146,42 +136,129 @@ GIT_EXTERN(unsigned int) git_commit_parentcount(git_commit *commit); */ GIT_EXTERN(int) git_commit_parent(git_commit **parent, git_commit *commit, unsigned int n); + /** - * Add a new parent commit to an existing commit - * @param commit the commit object - * @param new_parent the new commit which will be a parent + * Create a new commit in the repository + * + * + * @param oid Pointer where to store the OID of the + * newly created commit + * + * @param repo Repository where to store the commit + * + * @param update_ref If not NULL, name of the reference that + * will be updated to point to this commit. If the reference + * is not direct, it will be resolved to a direct reference. + * Use "HEAD" to update the HEAD of the current branch and + * make it point to this commit + * + * @param author Signature representing the author and the authory + * time of this commit + * + * @param committer Signature representing the committer and the + * commit time of this commit + * + * @param message Full message for this commit + * + * @param tree_oid Object ID of the tree for this commit. Note that + * no validation is performed on this OID. Use the _o variants of + * this method to assure a proper tree is passed to the commit. + * + * @param parent_count Number of parents for this commit + * + * @param parents Array of pointers to parent OIDs for this commit. + * Note that no validation is performed on these OIDs. Use the _o + * variants of this method to assure that are parents for the commit + * are proper objects. + * * @return 0 on success; error code otherwise + * The created commit will be written to the Object Database and + * the given reference will be updated to point to it */ -GIT_EXTERN(int) git_commit_add_parent(git_commit *commit, git_commit *new_parent); +GIT_EXTERN(int) git_commit_create( + git_oid *oid, + git_repository *repo, + const char *update_ref, + const git_signature *author, + const git_signature *committer, + const char *message, + const git_oid *tree_oid, + int parent_count, + const git_oid *parent_oids[]); /** - * Set the message of a commit - * @param commit the commit object - * @param message the new message + * Create a new commit in the repository using `git_object` + * instances as parameters. + * + * The `tree_oid` and `parent_oids` paremeters now take a instance + * of `git_tree` and `git_commit`, respectively. + * + * All other parameters remain the same + * + * @see git_commit_create */ -GIT_EXTERN(void) git_commit_set_message(git_commit *commit, const char *message); +GIT_EXTERN(int) git_commit_create_o( + git_oid *oid, + git_repository *repo, + const char *update_ref, + const git_signature *author, + const git_signature *committer, + const char *message, + const git_tree *tree, + int parent_count, + const git_commit *parents[]); /** - * Set the committer of a commit - * @param commit the commit object - * @param author_sig signature of the committer + * Create a new commit in the repository using `git_object` + * instances and a variable argument list. + * + * The `tree_oid` paremeter now takes a instance + * of `const git_tree *`. + * + * The parents for the commit are specified as a variable + * list of pointers to `const git_commit *`. Note that this + * is a convenience method which may not be safe to export + * for certain languages or compilers + * + * All other parameters remain the same + * + * @see git_commit_create */ -GIT_EXTERN(void) git_commit_set_committer(git_commit *commit, const git_signature *committer_sig); +GIT_EXTERN(int) git_commit_create_ov( + git_oid *oid, + git_repository *repo, + const char *update_ref, + const git_signature *author, + const git_signature *committer, + const char *message, + const git_tree *tree, + int parent_count, + ...); + /** - * Set the author of a commit - * @param commit the commit object - * @param author_sig signature of the author + * Create a new commit in the repository using + * a variable argument list. + * + * The parents for the commit are specified as a variable + * list of pointers to `const git_oid *`. Note that this + * is a convenience method which may not be safe to export + * for certain languages or compilers + * + * All other parameters remain the same + * + * @see git_commit_create */ -GIT_EXTERN(void) git_commit_set_author(git_commit *commit, const git_signature *author_sig); - -/** - * Set the tree which is pointed to by a commit - * @param commit the commit object - * @param tree the new tree - * @param 0 on success; error code otherwise - */ -GIT_EXTERN(int) git_commit_set_tree(git_commit *commit, git_tree *tree); +GIT_EXTERN(int) git_commit_create_v( + git_oid *oid, + git_repository *repo, + const char *update_ref, + const git_signature *author, + const git_signature *committer, + const char *message, + const git_oid *tree_oid, + int parent_count, + ...); /** @} */ GIT_END_DECL diff --git a/include/git2/common.h b/include/git2/common.h index f9045210c..7cfb8982e 100644 --- a/include/git2/common.h +++ b/include/git2/common.h @@ -158,6 +158,9 @@ /** The state of the reference is not valid */ #define GIT_EINVALIDREFSTATE (GIT_ERROR - 21) +/** This feature has not been implemented yet */ +#define GIT_ENOTIMPLEMENTED (GIT_ERROR - 22) + GIT_BEGIN_DECL typedef struct { diff --git a/include/git2/object.h b/include/git2/object.h index 748386f69..16dde8e56 100644 --- a/include/git2/object.h +++ b/include/git2/object.h @@ -42,7 +42,8 @@ GIT_BEGIN_DECL * Lookup a reference to one of the objects in a repostory. * * The generated reference is owned by the repository and - * should not be freed by the user. + * should be closed with the `git_object_close` method + * instead of free'd manually. * * The 'type' parameter must match the type of the object * in the odb; the method will fail otherwise. @@ -57,55 +58,9 @@ GIT_BEGIN_DECL */ GIT_EXTERN(int) git_object_lookup(git_object **object, git_repository *repo, const git_oid *id, git_otype type); -/** - * Create a new in-memory repository object with - * the given type. - * - * The object's attributes can be filled in using the - * corresponding setter methods. - * - * The object will be written back to given git_repository - * when the git_object_write() function is called; objects - * cannot be written to disk until all their main - * attributes have been properly filled. - * - * Objects are instantiated with no SHA1 id; their id - * will be automatically generated when writing to the - * repository. - * - * @param object pointer to the new object - * @parem repo Repository where the object belongs - * @param type Type of the object to be created - * @return the new object - */ -GIT_EXTERN(int) git_object_new(git_object **object, git_repository *repo, git_otype type); - - -/** - * Write back an object to disk. - * - * The object will be written to its corresponding - * repository. - * - * If the object has no changes since it was first - * read from the repository, no actions will take place. - * - * If the object has been modified since it was read from - * the repository, or it has been created from scratch - * in memory, it will be written to the repository and - * its SHA1 ID will be updated accordingly. - * - * @param object Git object to write back - * @return 0 on success; otherwise an error code - */ -GIT_EXTERN(int) git_object_write(git_object *object); - /** * Get the id (SHA1) of a repository object * - * In-memory objects created by git_object_new() do not - * have a SHA1 ID until they are written on a repository. - * * @param obj the repository object * @return the SHA1 id */ @@ -137,14 +92,8 @@ GIT_EXTERN(git_repository *) git_object_owner(const git_object *obj); * by the repository. * * IMPORTANT: - * It is *not* necessary to call this method when you stop using - * an object, since all object memory is automatically reclaimed - * by the repository when it is freed. - * - * Forgetting to call `git_object_close` does not cause memory - * leaks, but it's is recommended to close as soon as possible - * the biggest objects (e.g. blobs) to prevent wasting memory - * space. + * It *is* necessary to call this method when you stop using + * an object. Failure to do so will cause a memory leak. * * @param object the object to close */ diff --git a/include/git2/odb.h b/include/git2/odb.h index 0d285897c..8926b446c 100644 --- a/include/git2/odb.h +++ b/include/git2/odb.h @@ -28,6 +28,7 @@ #include "common.h" #include "types.h" #include "oid.h" +#include "odb_backend.h" /** * @file git2/odb.h @@ -100,61 +101,49 @@ GIT_EXTERN(int) git_odb_add_alternate(git_odb *odb, git_odb_backend *backend, in /** * Close an open object database. + * * @param db database pointer to close. If NULL no action is taken. */ GIT_EXTERN(void) git_odb_close(git_odb *db); -/** An object read from the database. */ -typedef struct { - void *data; /**< Raw, decompressed object data. */ - size_t len; /**< Total number of bytes in data. */ - git_otype type; /**< Type of this object. */ -} git_rawobj; - /** * Read an object from the database. * - * If GIT_ENOTFOUND then out->data is set to NULL. + * This method queries all avaiable ODB backends + * trying to read the given OID. * - * @param out object descriptor to populate upon reading. + * The returned object is reference counted and + * internally cached, so it should be closed + * by the user once it's no longer in use. + * + * @param out pointer where to store the read object * @param db database to search for the object in. * @param id identity of the object to read. * @return * - GIT_SUCCESS if the object was read; * - GIT_ENOTFOUND if the object is not in the database. */ -GIT_EXTERN(int) git_odb_read(git_rawobj *out, git_odb *db, const git_oid *id); +GIT_EXTERN(int) git_odb_read(git_odb_object **out, git_odb *db, const git_oid *id); /** * Read the header of an object from the database, without * reading its full contents. * - * Only the 'type' and 'len' fields of the git_rawobj structure - * are filled. The 'data' pointer will always be NULL. + * The header includes the length and the type of an object. * - * The raw object pointed by 'out' doesn't need to be manually - * closed with git_rawobj_close(). + * Note that most backends do not support reading only the header + * of an object, so the whole object will be read and then the + * header will be returned. * - * @param out object descriptor to populate upon reading. + * @param len_p pointer where to store the length + * @param type_p pointer where to store the type * @param db database to search for the object in. * @param id identity of the object to read. * @return * - GIT_SUCCESS if the object was read; * - GIT_ENOTFOUND if the object is not in the database. */ -GIT_EXTERN(int) git_odb_read_header(git_rawobj *out, git_odb *db, const git_oid *id); - -/** - * Write an object to the database. - * - * @param id identity of the object written. - * @param db database to which the object should be written. - * @param obj object descriptor for the object to write. - * @return - * - GIT_SUCCESS if the object was written; - * - GIT_ERROR otherwise. - */ -GIT_EXTERN(int) git_odb_write(git_oid *id, git_odb *db, git_rawobj *obj); +GIT_EXTERN(int) git_odb_read_header(size_t *len_p, git_otype *type_p, git_odb *db, const git_oid *id); /** * Determine if the given object can be found in the object database. @@ -162,39 +151,89 @@ GIT_EXTERN(int) git_odb_write(git_oid *id, git_odb *db, git_rawobj *obj); * @param db database to be searched for the given object. * @param id the object to search for. * @return - * - true, if the object was found - * - false, otherwise + * - 1, if the object was found + * - 0, otherwise */ GIT_EXTERN(int) git_odb_exists(git_odb *db, const git_oid *id); - - - +/** + * Open a stream to write an object into the ODB + * + * The type and final length of the object must be specified + * when opening the stream. + * + * The returned stream will be of type `GIT_STREAM_WRONLY` and + * will have the following methods: + * + * - stream->write: write `n` bytes into the stream + * - stream->finalize_write: close the stream and store the object in + * the odb + * - stream->free: free the stream + * + * The streaming write won't be effective until `stream->finalize_write` + * is called and returns without an error + * + * The stream must always be free'd or will leak memory. + * + * @see git_odb_stream + * + * @param stream pointer where to store the stream + * @param db object database where the stream will write + * @param size final size of the object that will be written + * @para type type of the object that will be written + * @return 0 if the stream was created; error code otherwise + */ +GIT_EXTERN(int) git_odb_open_wstream(git_odb_stream **stream, git_odb *db, size_t size, git_otype type); /** - * Determine the object-ID (sha1 hash) of the given git_rawobj. + * Open a stream to read an object from the ODB * - * The input obj must be a valid loose object type and the data - * pointer must not be NULL, unless the len field is also zero. + * Note that most backends do *not* support streaming reads + * because they store their objects as compressed/delta'ed blobs. + * + * It's recommended to use `git_odb_read` instead, which is + * assured to work on all backends. + * + * The returned stream will be of type `GIT_STREAM_RDONLY` and + * will have the following methods: + * + * - stream->read: read `n` bytes from the stream + * - stream->free: free the stream + * + * The stream must always be free'd or will leak memory. + * + * @see git_odb_stream + * + * @param stream pointer where to store the stream + * @param db object database where the stream will read from + * @param oid oid of the object the stream will read from + * @return 0 if the stream was created; error code otherwise + */ +GIT_EXTERN(int) git_odb_open_rstream(git_odb_stream **stream, git_odb *db, const git_oid *oid); + +/** + * Determine the object-ID (sha1 hash) of a data buffer + * + * The resulting SHA-1 OID will the itentifier for the data + * buffer as if the data buffer it were to written to the ODB. * * @param id the resulting object-ID. - * @param obj the object whose hash is to be determined. - * @return - * - GIT_SUCCESS if the object-ID was correctly determined. - * - GIT_ERROR if the given object is malformed. + * @param data data to hash + * @param len size of the data + * @param type of the data to hash + * @return 0 on success; error code otherwise */ -GIT_EXTERN(int) git_rawobj_hash(git_oid *id, git_rawobj *obj); +GIT_EXTERN(int) git_odb_hash(git_oid *id, const void *data, size_t len, git_otype type); /** - * Release all memory used by the obj structure. + * Close an ODB object * - * As a result of this call, obj->data will be set to NULL. + * This method must always be called once a `git_odb_object` is no + * longer needed, otherwise memory will leak. * - * If obj->data is already NULL, nothing happens. - * - * @param obj object descriptor to free. + * @param object object to close */ -GIT_EXTERN(void) git_rawobj_close(git_rawobj *obj); +GIT_EXTERN(void) git_odb_object_close(git_odb_object *object); /** @} */ GIT_END_DECL diff --git a/include/git2/odb_backend.h b/include/git2/odb_backend.h index 0e817eb37..3875ec7f6 100644 --- a/include/git2/odb_backend.h +++ b/include/git2/odb_backend.h @@ -39,24 +39,32 @@ */ GIT_BEGIN_DECL +struct git_odb_stream; + /** An instance for a custom backend */ struct git_odb_backend { git_odb *odb; int (* read)( - git_rawobj *, + void **, size_t *, git_otype *, struct git_odb_backend *, const git_oid *); int (* read_header)( - git_rawobj *, + size_t *, git_otype *, struct git_odb_backend *, const git_oid *); - int (* write)( - git_oid *id, + int (* writestream)( + struct git_odb_stream **, struct git_odb_backend *, - git_rawobj *obj); + size_t, + git_otype); + + int (* readstream)( + struct git_odb_stream **, + struct git_odb_backend *, + const git_oid *); int (* exists)( struct git_odb_backend *, @@ -65,12 +73,28 @@ struct git_odb_backend { void (* free)(struct git_odb_backend *); }; +/** A stream to read/write from a backend */ +struct git_odb_stream { + struct git_odb_backend *backend; + int mode; + + int (*read)(struct git_odb_stream *stream, char *buffer, size_t len); + int (*write)(struct git_odb_stream *stream, const char *buffer, size_t len); + int (*finalize_write)(git_oid *oid_p, struct git_odb_stream *stream); + void (*free)(struct git_odb_stream *stream); +}; + +/** Streaming mode */ +typedef enum { + GIT_STREAM_RDONLY = (1 << 1), + GIT_STREAM_WRONLY = (1 << 2), + GIT_STREAM_RW = (GIT_STREAM_RDONLY | GIT_STREAM_WRONLY), +} git_odb_streammode; + + GIT_EXTERN(int) git_odb_backend_pack(git_odb_backend **backend_out, const char *objects_dir); GIT_EXTERN(int) git_odb_backend_loose(git_odb_backend **backend_out, const char *objects_dir); - -#ifdef GIT2_SQLITE_BACKEND GIT_EXTERN(int) git_odb_backend_sqlite(git_odb_backend **backend_out, const char *sqlite_db); -#endif GIT_END_DECL diff --git a/include/git2/tag.h b/include/git2/tag.h index f1669eb90..c343f6bf4 100644 --- a/include/git2/tag.h +++ b/include/git2/tag.h @@ -41,8 +41,6 @@ GIT_BEGIN_DECL /** * Lookup a tag object from the repository. - * The generated tag object is owned by the revision - * repo and shall not be freed by the user. * * @param tag pointer to the looked up tag * @param repo the repo to use when locating the tag. @@ -54,24 +52,9 @@ GIT_INLINE(int) git_tag_lookup(git_tag **tag, git_repository *repo, const git_oi return git_object_lookup((git_object **)tag, repo, id, (git_otype)GIT_OBJ_TAG); } -/** - * Create a new in-memory git_tag. - * - * The tag object must be manually filled using - * setter methods before it can be written to its - * repository. - * - * @param tag pointer to the new tag - * @param repo The repository where the object will reside - * @return 0 on success; error code otherwise - */ -GIT_INLINE(int) git_tag_new(git_tag **tag, git_repository *repo) -{ - return git_object_new((git_object **)tag, repo, (git_otype)GIT_OBJ_TAG); -} - /** * Get the id of a tag. + * * @param tag a previously loaded tag. * @return object identity for the tag. */ @@ -79,6 +62,10 @@ GIT_EXTERN(const git_oid *) git_tag_id(git_tag *tag); /** * Get the tagged object of a tag + * + * This method performs a repository lookup for the + * given object and returns it + * * @param target pointer where to store the target * @param tag a previously loaded tag. * @return 0 on success; error code otherwise @@ -87,6 +74,7 @@ GIT_EXTERN(int) git_tag_target(git_object **target, git_tag *t); /** * Get the OID of the tagged object of a tag + * * @param tag a previously loaded tag. * @return pointer to the OID */ @@ -94,6 +82,7 @@ GIT_EXTERN(const git_oid *) git_tag_target_oid(git_tag *t); /** * Get the type of a tag's tagged object + * * @param tag a previously loaded tag. * @return type of the tagged object */ @@ -101,6 +90,7 @@ GIT_EXTERN(git_otype) git_tag_type(git_tag *t); /** * Get the name of a tag + * * @param tag a previously loaded tag. * @return name of the tag */ @@ -108,6 +98,7 @@ GIT_EXTERN(const char *) git_tag_name(git_tag *t); /** * Get the tagger (author) of a tag + * * @param tag a previously loaded tag. * @return reference to the tag's author */ @@ -115,39 +106,69 @@ GIT_EXTERN(const git_signature *) git_tag_tagger(git_tag *t); /** * Get the message of a tag + * * @param tag a previously loaded tag. * @return message of the tag */ GIT_EXTERN(const char *) git_tag_message(git_tag *t); -/** - * Set the target of a tag (i.e. the object that the tag points to) - * @param tag The tag to modify - * @param target the new tagged target - */ -GIT_EXTERN(int) git_tag_set_target(git_tag *tag, git_object *target); /** - * Set the name of a tag - * @param tag The tag to modify - * @param name the new name for the tag + * Create a new tag in the repository from an OID + * + * @param oid Pointer where to store the OID of the + * newly created tag + * + * @param repo Repository where to store the tag + * + * @param tag_name Name for the tag; this name is validated + * for consistency + * + * @param target OID to which this tag points; note that no + * validation is done on this OID. Use the _o version of this + * method to assure a proper object is being tagged + * + * @param target_type Type of the tagged OID; note that no + * validation is performed here either + * + * @param tagger Signature of the tagger for this tag, and + * of the tagging time + * + * @param message Full message for this tag + * + * @return 0 on success; error code otherwise. + * A tag object is written to the ODB, and a proper reference + * is written in the /refs/tags folder, pointing to it */ -GIT_EXTERN(void) git_tag_set_name(git_tag *tag, const char *name); +GIT_EXTERN(int) git_tag_create( + git_oid *oid, + git_repository *repo, + const char *tag_name, + const git_oid *target, + git_otype target_type, + const git_signature *tagger, + const char *message); + /** - * Set the tagger of a tag - * @param tag The tag to modify - * @param tagger_sig signature of the tagging action - * @return 0 on success; error code otherwise + * Create a new tag in the repository from an existing + * `git_object` instance + * + * This method replaces the `target` and `target_type` + * paremeters of `git_tag_create` by a single instance + * of a `const git_object *`, which is assured to be + * a proper object in the ODB and hence will create + * a valid tag + * + * @see git_tag_create */ -GIT_EXTERN(void) git_tag_set_tagger(git_tag *tag, const git_signature *tagger_sig); - -/** - * Set the message of a tag - * @param tag The tag to modify - * @param message the new tagger for the tag - */ -GIT_EXTERN(void) git_tag_set_message(git_tag *tag, const char *message); +GIT_EXTERN(int) git_tag_create_o( + git_oid *oid, + git_repository *repo, + const char *tag_name, + const git_object *target, + const git_signature *tagger, + const char *message); /** @} */ GIT_END_DECL diff --git a/include/git2/tree.h b/include/git2/tree.h index 3085b3fd6..ec2b51646 100644 --- a/include/git2/tree.h +++ b/include/git2/tree.h @@ -41,8 +41,6 @@ GIT_BEGIN_DECL /** * Lookup a tree object from the repository. - * The generated tree object is owned by the revision - * repo and shall not be freed by the user. * * @param tree pointer to the looked up tree * @param repo the repo to use when locating the tree. @@ -54,32 +52,17 @@ GIT_INLINE(int) git_tree_lookup(git_tree **tree, git_repository *repo, const git return git_object_lookup((git_object **)tree, repo, id, GIT_OBJ_TREE); } -/** - * Create a new in-memory git_tree. - * - * The tree object must be manually filled using - * setter methods before it can be written to its - * repository. - * - * @param tree pointer to the new tree - * @param repo The repository where the object will reside - * @return 0 on success; error code otherwise - */ -GIT_INLINE(int) git_tree_new(git_tree **tree, git_repository *repo) -{ - return git_object_new((git_object **)tree, repo, GIT_OBJ_TREE); -} - /** * Get the id of a tree. + * * @param tree a previously loaded tree. * @return object identity for the tree. */ GIT_EXTERN(const git_oid *) git_tree_id(git_tree *tree); - /** * Get the number of entries listed in a tree + * * @param tree a previously loaded tree. * @return the number of entries in the tree */ @@ -87,6 +70,7 @@ GIT_EXTERN(size_t) git_tree_entrycount(git_tree *tree); /** * Lookup a tree entry by its filename + * * @param tree a previously loaded tree. * @param filename the filename of the desired entry * @return the tree entry; NULL if not found @@ -95,6 +79,7 @@ GIT_EXTERN(git_tree_entry *) git_tree_entry_byname(git_tree *tree, const char *f /** * Lookup a tree entry by its position in the tree + * * @param tree a previously loaded tree. * @param idx the position in the entry list * @return the tree entry; NULL if not found @@ -103,6 +88,7 @@ GIT_EXTERN(git_tree_entry *) git_tree_entry_byindex(git_tree *tree, int idx); /** * Get the UNIX file attributes of a tree entry + * * @param entry a tree entry * @return attributes as an integer */ @@ -110,6 +96,7 @@ GIT_EXTERN(unsigned int) git_tree_entry_attributes(git_tree_entry *entry); /** * Get the filename of a tree entry + * * @param entry a tree entry * @return the name of the file */ @@ -117,6 +104,7 @@ GIT_EXTERN(const char *) git_tree_entry_name(git_tree_entry *entry); /** * Get the id of the object pointed by the entry + * * @param entry a tree entry * @return the oid of the object */ @@ -126,97 +114,11 @@ GIT_EXTERN(const git_oid *) git_tree_entry_id(git_tree_entry *entry); * Convert a tree entry to the git_object it points too. * * @param object pointer to the converted object + * @param repo repository where to lookup the pointed object * @param entry a tree entry * @return a reference to the pointed object in the repository */ -GIT_EXTERN(int) git_tree_entry_2object(git_object **object, git_tree_entry *entry); - -/** - * Add a new entry to a tree and return the new entry. - * - * This will mark the tree as modified; the new entry will - * be written back to disk on the next git_object_write() - * - * @param entry_out Pointer to the entry that just got - * created. May be NULL if you are not interested on - * getting the new entry - * @param tree Tree object to store the entry - * @iparam id OID for the tree entry - * @param filename Filename for the tree entry - * @param attributes UNIX file attributes for the entry - * @return 0 on success; otherwise error code - */ -GIT_EXTERN(int) git_tree_add_entry(git_tree_entry **entry_out, git_tree *tree, const git_oid *id, const char *filename, int attributes); - -/** - * Remove an entry by its index. - * - * Index must be >= 0 and < than git_tree_entrycount(). - * - * This will mark the tree as modified; the modified entry will - * be written back to disk on the next git_object_write() - * - * @param tree Tree where to remove the entry - * @param idx index of the entry - * @return 0 on successful removal; GIT_ENOTFOUND if the entry wasn't found - */ -GIT_EXTERN(int) git_tree_remove_entry_byindex(git_tree *tree, int idx); - -/** - * Remove an entry by its filename. - * - * This will mark the tree as modified; the modified entry will - * be written back to disk on the next git_object_write() - * - * @param tree Tree where to remove the entry - * @param filename File name of the entry - * @return 0 on successful removal; GIT_ENOTFOUND if the entry wasn't found - */ -GIT_EXTERN(int) git_tree_remove_entry_byname(git_tree *tree, const char *filename); - -/** - * Clear all the entries in a tree. - * - * This will mark the tree as modified; the modified entry will - * be written back to disk on the next git_object_write(). - * - * @param tree Tree object whose entries are to be sorted - */ -GIT_EXTERN(void) git_tree_clear_entries(git_tree *tree); - -/** - * Change the SHA1 id of a tree entry. - * - * This will mark the tree that contains the entry as modified; - * the modified entry will be written back to disk on the next git_object_write() - * - * @param entry Entry object which will be modified - * @param oid new SHA1 oid for the entry - */ -GIT_EXTERN(void) git_tree_entry_set_id(git_tree_entry *entry, const git_oid *oid); - -/** - * Change the filename of a tree entry. - * - * This will mark the tree that contains the entry as modified; - * the modified entry will be written back to disk on the next git_object_write() - * - * @param entry Entry object which will be modified - * @param oid new filename for the entry - */ -GIT_EXTERN(void) git_tree_entry_set_name(git_tree_entry *entry, const char *name); - -/** - * Change the attributes of a tree entry. - * - * This will mark the tree that contains the entry as modified; - * the modified entry will be written back to disk on the next git_object_write() - * - * @param entry Entry object which will be modified - * @param oid new attributes for the entry - * @return 0 if the attributes were properly set; error code otherwise - */ -GIT_EXTERN(int) git_tree_entry_set_attributes(git_tree_entry *entry, unsigned int attr); +GIT_EXTERN(int) git_tree_entry_2object(git_object **object_out, git_repository *repo, git_tree_entry *entry); /** @} */ GIT_END_DECL diff --git a/include/git2/types.h b/include/git2/types.h index b5a8d7b2d..64f7fc72e 100644 --- a/include/git2/types.h +++ b/include/git2/types.h @@ -71,7 +71,6 @@ typedef time_t git_time_t; #endif - /** Basic type (loose or packed) of any Git object. */ typedef enum { GIT_OBJ_ANY = -2, /**< Object can be any of the following */ @@ -92,6 +91,12 @@ typedef struct git_odb git_odb; /** A custom backend in an ODB */ typedef struct git_odb_backend git_odb_backend; +/** An object read from the ODB */ +typedef struct git_odb_object git_odb_object; + +/** A stream to read/write from the ODB */ +typedef struct git_odb_stream git_odb_stream; + /** * Representation of an existing git repository, * including all its object contents diff --git a/src/backends/sqlite.c b/src/backends/sqlite.c index b4c941a59..72d7b4d8e 100644 --- a/src/backends/sqlite.c +++ b/src/backends/sqlite.c @@ -272,4 +272,13 @@ cleanup: return GIT_ERROR; } +#else + +int git_odb_backend_sqlite(git_odb_backend **GIT_UNUSED(backend_out), const char *GIT_UNUSED(sqlite_db)) +{ + GIT_UNUSED_ARG(backend_out); + GIT_UNUSED_ARG(sqlite_db); + return GIT_ENOTIMPLEMENTED; +} + #endif /* HAVE_SQLITE3 */ diff --git a/src/blob.c b/src/blob.c index 1e03b6b67..bc0a08a8a 100644 --- a/src/blob.c +++ b/src/blob.c @@ -33,104 +33,89 @@ const void *git_blob_rawcontent(git_blob *blob) { assert(blob); - - if (blob->content.data != NULL) - return blob->content.data; - - if (blob->object.in_memory) - return NULL; - - if (!blob->object.source.open && git_object__source_open((git_object *)blob) < GIT_SUCCESS) - return NULL; - - return blob->object.source.raw.data; + return blob->odb_object->raw.data; } int git_blob_rawsize(git_blob *blob) { assert(blob); - - if (blob->content.data != NULL) - return blob->content.len; - - return blob->object.source.raw.len; + return blob->odb_object->raw.len; } void git_blob__free(git_blob *blob) { - gitfo_free_buf(&blob->content); + git_odb_object_close(blob->odb_object); free(blob); } -int git_blob__parse(git_blob *blob) +int git_blob__parse(git_blob *blob, git_odb_object *odb_obj) { assert(blob); + git_cached_obj_incref((git_cached_obj *)odb_obj); + blob->odb_object = odb_obj; return GIT_SUCCESS; } -int git_blob__writeback(git_blob *blob, git_odb_source *src) -{ - assert(blob->object.modified); - - if (blob->content.data == NULL) - return GIT_EMISSINGOBJDATA; - - return git__source_write(src, blob->content.data, blob->content.len); -} - -int git_blob_set_rawcontent(git_blob *blob, const void *buffer, size_t len) -{ - assert(blob && buffer); - - blob->object.modified = 1; - - git_object__source_close((git_object *)blob); - - if (blob->content.data != NULL) - gitfo_free_buf(&blob->content); - - blob->content.data = git__malloc(len); - blob->content.len = len; - - if (blob->content.data == NULL) - return GIT_ENOMEM; - - memcpy(blob->content.data, buffer, len); - - return GIT_SUCCESS; -} - -int git_blob_set_rawcontent_fromfile(git_blob *blob, const char *filename) -{ - assert(blob && filename); - blob->object.modified = 1; - - if (blob->content.data != NULL) - gitfo_free_buf(&blob->content); - - return gitfo_read_file(&blob->content, filename); -} - -int git_blob_writefile(git_oid *written_id, git_repository *repo, const char *path) +int git_blob_create_frombuffer(git_oid *oid, git_repository *repo, const void *buffer, size_t len) { int error; - git_blob *blob; + git_odb_stream *stream; - if (gitfo_exists(path) < 0) - return GIT_ENOTFOUND; - - if ((error = git_blob_new(&blob, repo)) < GIT_SUCCESS) + if ((error = git_odb_open_wstream(&stream, repo->db, len, GIT_OBJ_BLOB)) < GIT_SUCCESS) return error; - if ((error = git_blob_set_rawcontent_fromfile(blob, path)) < GIT_SUCCESS) - return error; + stream->write(stream, buffer, len); - if ((error = git_object_write((git_object *)blob)) < GIT_SUCCESS) - return error; + error = stream->finalize_write(oid, stream); + stream->free(stream); - git_oid_cpy(written_id, git_object_id((git_object *)blob)); - - git_object_close((git_object*)blob); - return GIT_SUCCESS; + return error; +} + +int git_blob_create_fromfile(git_oid *oid, git_repository *repo, const char *path) +{ + int error, fd; + char full_path[GIT_PATH_MAX]; + char buffer[2048]; + git_off_t size; + git_odb_stream *stream; + + if (repo->path_workdir == NULL) + return GIT_ENOTFOUND; + + git__joinpath(full_path, repo->path_workdir, path); + + if ((fd = gitfo_open(full_path, O_RDONLY)) < 0) + return GIT_ENOTFOUND; + + if ((size = gitfo_size(fd)) < 0 || !git__is_sizet(size)) { + gitfo_close(fd); + return GIT_EOSERR; + } + + if ((error = git_odb_open_wstream(&stream, repo->db, (size_t)size, GIT_OBJ_BLOB)) < GIT_SUCCESS) { + gitfo_close(fd); + return error; + } + + while (size > 0) { + ssize_t read_len; + + read_len = read(fd, buffer, sizeof(buffer)); + + if (read_len < 0) { + gitfo_close(fd); + stream->free(stream); + return GIT_EOSERR; + } + + stream->write(stream, buffer, read_len); + size -= read_len; + } + + error = stream->finalize_write(oid, stream); + stream->free(stream); + + return error; } diff --git a/src/blob.h b/src/blob.h index febc296fe..4300d7e54 100644 --- a/src/blob.h +++ b/src/blob.h @@ -3,15 +3,15 @@ #include "git2/blob.h" #include "repository.h" +#include "odb.h" #include "fileops.h" struct git_blob { git_object object; - gitfo_buf content; + git_odb_object *odb_object; }; void git_blob__free(git_blob *blob); -int git_blob__parse(git_blob *blob); -int git_blob__writeback(git_blob *blob, git_odb_source *src); +int git_blob__parse(git_blob *blob, git_odb_object *obj); #endif diff --git a/src/cache.c b/src/cache.c index 5a9b8415c..fdf5c1b80 100644 --- a/src/cache.c +++ b/src/cache.c @@ -32,23 +32,6 @@ #define GIT_CACHE_OPENADR 3 -GIT_INLINE(int) cached_obj_compare(git_cached_obj *obj, const git_oid *oid) -{ - return git_oid_cmp(&obj->oid, oid); -} - -GIT_INLINE(void) cached_obj_incref(git_cached_obj *obj) -{ - git_atomic_inc(&obj->refcount); -} - -GIT_INLINE(void) cached_obj_decref(git_cached_obj *obj, git_cached_obj_freeptr free_obj) -{ - if (git_atomic_dec(&obj->refcount) == 0) - free_obj(obj); -} - - void git_cache_init(git_cache *cache, size_t size, git_cached_obj_freeptr free_ptr) { size_t i; @@ -82,7 +65,9 @@ void git_cache_free(git_cache *cache) size_t i; for (i = 0; i < (cache->size_mask + 1); ++i) { - cached_obj_decref(cache->nodes[i].ptr, cache->free_obj); + if (cache->nodes[i].ptr) + git_cached_obj_decref(cache->nodes[i].ptr, cache->free_obj); + git_mutex_free(&cache->nodes[i].lock); } @@ -103,8 +88,8 @@ void *git_cache_get(git_cache *cache, const git_oid *oid) git_mutex_lock(&node->lock); { - if (cached_obj_compare(node->ptr, oid) == 0) { - cached_obj_incref(node->ptr); + if (node->ptr && git_cached_obj_compare(node->ptr, oid) == 0) { + git_cached_obj_incref(node->ptr); node->lru = ++cache->lru_count; found = 1; } @@ -121,14 +106,14 @@ void *git_cache_try_store(git_cache *cache, void *entry) cache_node *nodes[GIT_CACHE_OPENADR], *lru_node; const uint32_t *hash; const git_oid *oid; - size_t i, stored = 0; + size_t i; oid = &((git_cached_obj*)entry)->oid; hash = (const uint32_t *)oid->id; /* increase the refcount on this object, because * the cache now owns it */ - cached_obj_incref(entry); + git_cached_obj_incref(entry); for (i = 0; i < GIT_CACHE_OPENADR; ++i) { size_t pos = hash[i] & cache->size_mask; @@ -139,34 +124,35 @@ void *git_cache_try_store(git_cache *cache, void *entry) lru_node = nodes[0]; - for (i = 0; !stored && entry && i < GIT_CACHE_OPENADR; ++i) { + for (i = 0; i < GIT_CACHE_OPENADR; ++i) { if (nodes[i]->ptr == NULL) { nodes[i]->ptr = entry; nodes[i]->lru = ++cache->lru_count; - stored = 1; - } else if (cached_obj_compare(nodes[i]->ptr, oid) == 0) { - cached_obj_decref(entry, cache->free_obj); + break; + } else if (git_cached_obj_compare(nodes[i]->ptr, oid) == 0) { + git_cached_obj_decref(entry, cache->free_obj); entry = nodes[i]->ptr; - stored = 1; + nodes[i]->lru = ++cache->lru_count; + break; } if (nodes[i]->lru < lru_node->lru) lru_node = nodes[i]; } - if (!stored) { + if (i == GIT_CACHE_OPENADR) { void *old_entry = lru_node->ptr; assert(old_entry); - cached_obj_decref(old_entry, cache->free_obj); + git_cached_obj_decref(old_entry, cache->free_obj); lru_node->ptr = entry; lru_node->lru = ++cache->lru_count; } /* increase the refcount again, because we are * returning it to the user */ - cached_obj_incref(entry); + git_cached_obj_incref(entry); for (i = 0; i < GIT_CACHE_OPENADR; ++i) git_mutex_unlock(&nodes[i]->lock); diff --git a/src/cache.h b/src/cache.h index 9f525e68c..975aaff7e 100644 --- a/src/cache.h +++ b/src/cache.h @@ -7,6 +7,8 @@ #include "thread-utils.h" +#define GIT_DEFAULT_CACHE_SIZE 128 + typedef void (*git_cached_obj_freeptr)(void *); typedef struct { @@ -35,4 +37,23 @@ void git_cache_free(git_cache *cache); void *git_cache_try_store(git_cache *cache, void *entry); void *git_cache_get(git_cache *cache, const git_oid *oid); + +GIT_INLINE(int) git_cached_obj_compare(git_cached_obj *obj, const git_oid *oid) +{ + return git_oid_cmp(&obj->oid, oid); +} + +GIT_INLINE(void) git_cached_obj_incref(git_cached_obj *obj) +{ + git_atomic_inc(&obj->refcount); +} + +GIT_INLINE(void) git_cached_obj_decref(git_cached_obj *obj, git_cached_obj_freeptr free_obj) +{ + if (git_atomic_dec(&obj->refcount) == 0) + free_obj(obj); +} + + + #endif diff --git a/src/commit.c b/src/commit.c index 8da170cad..2670c2e0a 100644 --- a/src/commit.c +++ b/src/commit.c @@ -29,9 +29,12 @@ #include "git2/signature.h" #include "common.h" +#include "odb.h" #include "commit.h" #include "signature.h" +#include + #define COMMIT_BASIC_PARSE 0x0 #define COMMIT_FULL_PARSE 0x1 @@ -71,35 +74,165 @@ const git_oid *git_commit_id(git_commit *c) return git_object_id((git_object *)c); } -int git_commit__writeback(git_commit *commit, git_odb_source *src) + +int git_commit_create_v( + git_oid *oid, + git_repository *repo, + const char *update_ref, + const git_signature *author, + const git_signature *committer, + const char *message, + const git_oid *tree_oid, + int parent_count, + ...) { - unsigned int i; + va_list ap; + int i, error; + const git_oid **oids; - git__write_oid(src, "tree", &commit->tree_oid); + oids = git__malloc(parent_count * sizeof(git_oid *)); - for (i = 0; i < commit->parent_oids.length; ++i) { - git_oid *parent_oid; + va_start(ap, parent_count); + for (i = 0; i < parent_count; ++i) + oids[i] = va_arg(ap, const git_oid *); + va_end(ap); - parent_oid = git_vector_get(&commit->parent_oids, i); - git__write_oid(src, "parent", parent_oid); + error = git_commit_create( + oid, repo, update_ref, author, committer, message, + tree_oid, parent_count, oids); + + free(oids); + return error; +} + +int git_commit_create_ov( + git_oid *oid, + git_repository *repo, + const char *update_ref, + const git_signature *author, + const git_signature *committer, + const char *message, + const git_tree *tree, + int parent_count, + ...) +{ + va_list ap; + int i, error; + const git_oid **oids; + + oids = git__malloc(parent_count * sizeof(git_oid *)); + + va_start(ap, parent_count); + for (i = 0; i < parent_count; ++i) + oids[i] = git_object_id(va_arg(ap, const git_object *)); + va_end(ap); + + error = git_commit_create( + oid, repo, update_ref, author, committer, message, + git_object_id((git_object *)tree), + parent_count, oids); + + free(oids); + return error; +} + +int git_commit_create_o( + git_oid *oid, + git_repository *repo, + const char *update_ref, + const git_signature *author, + const git_signature *committer, + const char *message, + const git_tree *tree, + int parent_count, + const git_commit *parents[]) +{ + int i, error; + const git_oid **oids; + + oids = git__malloc(parent_count * sizeof(git_oid *)); + + for (i = 0; i < parent_count; ++i) + oids[i] = git_object_id((git_object *)parents[i]); + + error = git_commit_create( + oid, repo, update_ref, author, committer, message, + git_object_id((git_object *)tree), + parent_count, oids); + + free(oids); + return error; +} + +int git_commit_create( + git_oid *oid, + git_repository *repo, + const char *update_ref, + const git_signature *author, + const git_signature *committer, + const char *message, + const git_oid *tree_oid, + int parent_count, + const git_oid *parents[]) +{ + size_t final_size = 0; + int message_length, author_length, committer_length; + + char *author_str, *committer_str; + + int error, i; + git_odb_stream *stream; + + message_length = strlen(message); + author_length = git_signature__write(&author_str, "author", author); + committer_length = git_signature__write(&committer_str, "committer", committer); + + if (author_length < 0 || committer_length < 0) + return GIT_ENOMEM; + + final_size += GIT_OID_LINE_LENGTH("tree"); + final_size += GIT_OID_LINE_LENGTH("parent") * parent_count; + final_size += author_length; + final_size += committer_length; + final_size += 1 + message_length; + + if ((error = git_odb_open_wstream(&stream, repo->db, final_size, GIT_OBJ_COMMIT)) < GIT_SUCCESS) + return error; + + git__write_oid(stream, "tree", tree_oid); + + for (i = 0; i < parent_count; ++i) + git__write_oid(stream, "parent", parents[i]); + + stream->write(stream, author_str, author_length); + free(author_str); + + stream->write(stream, committer_str, committer_length); + free(committer_str); + + + stream->write(stream, "\n", 1); + stream->write(stream, message, message_length); + + error = stream->finalize_write(oid, stream); + stream->free(stream); + + if (error == GIT_SUCCESS && update_ref != NULL) { + git_reference *head; + + error = git_reference_lookup(&head, repo, update_ref); + if (error < GIT_SUCCESS) + return error; + + if (git_reference_type(head) == GIT_REF_SYMBOLIC) { + if ((error = git_reference_resolve(&head, head)) < GIT_SUCCESS) + return error; + } + + error = git_reference_set_oid(head, oid); } - if (commit->author == NULL) - return GIT_EMISSINGOBJDATA; - - git_signature__write(src, "author", commit->author); - - if (commit->committer == NULL) - return GIT_EMISSINGOBJDATA; - - git_signature__write(src, "committer", commit->committer); - - if (commit->message != NULL) { - git__source_write(src, "\n", 1); - git__source_write(src, commit->message, strlen(commit->message)); - } - - return GIT_SUCCESS; + return error; } int commit_parse_buffer(git_commit *commit, void *data, size_t len) @@ -110,12 +243,7 @@ int commit_parse_buffer(git_commit *commit, void *data, size_t len) git_oid parent_oid; int error; - /* first parse; the vector hasn't been initialized yet */ - if (commit->parent_oids.contents == NULL) { - git_vector_init(&commit->parent_oids, 4, NULL); - } - - clear_parents(commit); + git_vector_init(&commit->parent_oids, 4, NULL); if ((error = git__parse_oid(&commit->tree_oid, &buffer, buffer_end, "tree ")) < GIT_SUCCESS) return error; @@ -134,17 +262,11 @@ int commit_parse_buffer(git_commit *commit, void *data, size_t len) return GIT_ENOMEM; } - if (commit->author) - git_signature_free(commit->author); - commit->author = git__malloc(sizeof(git_signature)); if ((error = git_signature__parse(commit->author, &buffer, buffer_end, "author ")) < GIT_SUCCESS) return error; /* Always parse the committer; we need the commit time */ - if (commit->committer) - git_signature_free(commit->committer); - commit->committer = git__malloc(sizeof(git_signature)); if ((error = git_signature__parse(commit->committer, &buffer, buffer_end, "committer ")) < GIT_SUCCESS) return error; @@ -176,11 +298,10 @@ int commit_parse_buffer(git_commit *commit, void *data, size_t len) return GIT_SUCCESS; } -int git_commit__parse(git_commit *commit) +int git_commit__parse(git_commit *commit, git_odb_object *obj) { - assert(commit && commit->object.source.open); - return commit_parse_buffer(commit, - commit->object.source.raw.data, commit->object.source.raw.len); + assert(commit); + return commit_parse_buffer(commit, obj->raw.data, obj->raw.len); } #define GIT_COMMIT_GETTER(_rvalue, _name, _return) \ @@ -218,82 +339,3 @@ int git_commit_parent(git_commit **parent, git_commit *commit, unsigned int n) } - -int git_commit_set_tree(git_commit *commit, git_tree *tree) -{ - const git_oid *oid; - - assert(commit && tree); - - if ((oid = git_object_id((git_object *)tree)) == NULL) - return GIT_EMISSINGOBJDATA; - - commit->object.modified = 1; - git_oid_cpy(&commit->tree_oid, oid); - return GIT_SUCCESS; -} - -int git_commit_add_parent(git_commit *commit, git_commit *new_parent) -{ - const git_oid *parent_oid; - git_oid *new_oid; - assert(commit && new_parent); - - if ((parent_oid = git_object_id((git_object *)new_parent)) == NULL) - return GIT_EMISSINGOBJDATA; - - new_oid = git__malloc(sizeof(git_oid)); - if (new_oid == NULL) - return GIT_ENOMEM; - - commit->object.modified = 1; - git_oid_cpy(new_oid, parent_oid); - return git_vector_insert(&commit->parent_oids, new_oid); -} - -void git_commit_set_author(git_commit *commit, const git_signature *author_sig) -{ - assert(commit && author_sig); - commit->object.modified = 1; - - git_signature_free(commit->author); - commit->author = git_signature_dup(author_sig); -} - -void git_commit_set_committer(git_commit *commit, const git_signature *committer_sig) -{ - assert(commit && committer_sig); - commit->object.modified = 1; - - git_signature_free(commit->committer); - commit->committer = git_signature_dup(committer_sig); -} - -void git_commit_set_message(git_commit *commit, const char *message) -{ - const char *line_end; - size_t message_len; - - commit->object.modified = 1; - - if (commit->message) - free(commit->message); - - if (commit->message_short) - free(commit->message_short); - - commit->message = git__strdup(message); - - /* Short message */ - if((line_end = strchr(message, '\n')) == NULL) { - commit->message_short = git__strdup(message); - return; - } - - message_len = line_end - message; - - commit->message_short = git__malloc(message_len + 1); - memcpy(commit->message_short, message, message_len); - commit->message_short[message_len] = 0; -} - diff --git a/src/commit.h b/src/commit.h index aaf349ca6..3d15c5044 100644 --- a/src/commit.h +++ b/src/commit.h @@ -22,8 +22,6 @@ struct git_commit { }; void git_commit__free(git_commit *c); -int git_commit__parse(git_commit *commit); - -int git_commit__writeback(git_commit *commit, git_odb_source *src); +int git_commit__parse(git_commit *commit, git_odb_object *obj); #endif diff --git a/src/delta-apply.h b/src/delta-apply.h index 642442de0..36c5cc60d 100644 --- a/src/delta-apply.h +++ b/src/delta-apply.h @@ -1,6 +1,8 @@ #ifndef INCLUDE_delta_apply_h__ #define INCLUDE_delta_apply_h__ +#include "odb.h" + /** * Apply a git binary delta to recover the original content. * diff --git a/src/errors.c b/src/errors.c index 880163f78..f6b964837 100644 --- a/src/errors.c +++ b/src/errors.c @@ -27,7 +27,8 @@ static struct { {GIT_EPACKEDREFSCORRUPTED, "The pack-refs file is either corrupted of its format is not currently supported"}, {GIT_EINVALIDPATH, "The path is invalid" }, {GIT_EREVWALKOVER, "The revision walker is empty; there are no more commits left to iterate"}, - {GIT_EINVALIDREFSTATE, "The state of the reference is not valid"} + {GIT_EINVALIDREFSTATE, "The state of the reference is not valid"}, + {GIT_ENOTIMPLEMENTED, "This feature has not been implemented yet"} }; const char *git_strerror(int num) diff --git a/src/filebuf.c b/src/filebuf.c index 4fc4f1486..607ad618d 100644 --- a/src/filebuf.c +++ b/src/filebuf.c @@ -77,43 +77,81 @@ void git_filebuf_cleanup(git_filebuf *file) if (file->fd >= 0) gitfo_close(file->fd); - if (gitfo_exists(file->path_lock) == GIT_SUCCESS) + if (file->path_lock && gitfo_exists(file->path_lock) == GIT_SUCCESS) gitfo_unlink(file->path_lock); if (file->digest) git_hash_free_ctx(file->digest); free(file->buffer); + free(file->z_buf); -#ifdef GIT_FILEBUF_THREADS - free(file->buffer_back); -#endif + deflateEnd(&file->zs); free(file->path_original); free(file->path_lock); } -static int flush_buffer(git_filebuf *file) +GIT_INLINE(int) flush_buffer(git_filebuf *file) { - int result = GIT_SUCCESS; + int result = file->write(file, file->buffer, file->buf_pos); + file->buf_pos = 0; + return result; +} - if (file->buf_pos > 0) { - result = gitfo_write(file->fd, file->buffer, file->buf_pos); +static int write_normal(git_filebuf *file, const void *source, size_t len) +{ + int result = 0; + + if (len > 0) { + result = gitfo_write(file->fd, (void *)source, len); if (file->digest) - git_hash_update(file->digest, file->buffer, file->buf_pos); - - file->buf_pos = 0; + git_hash_update(file->digest, source, len); } return result; } +static int write_deflate(git_filebuf *file, const void *source, size_t len) +{ + int result = Z_OK; + z_stream *zs = &file->zs; + + if (len > 0 || file->flush_mode == Z_FINISH) { + zs->next_in = (void *)source; + zs->avail_in = len; + + do { + int have; + + zs->next_out = file->z_buf; + zs->avail_out = file->buf_size; + + result = deflate(zs, file->flush_mode); + assert(result != Z_STREAM_ERROR); + + have = file->buf_size - zs->avail_out; + + if (gitfo_write(file->fd, file->z_buf, have) < GIT_SUCCESS) + return GIT_EOSERR; + + } while (zs->avail_out == 0); + + assert(zs->avail_in == 0); + + if (file->digest) + git_hash_update(file->digest, source, len); + } + + return GIT_SUCCESS; +} + int git_filebuf_open(git_filebuf *file, const char *path, int flags) { int error; size_t path_len; - if (file == NULL || path == NULL) + if (file == NULL) return GIT_ERROR; memset(file, 0x0, sizeof(git_filebuf)); @@ -122,37 +160,14 @@ int git_filebuf_open(git_filebuf *file, const char *path, int flags) file->buf_pos = 0; file->fd = -1; - path_len = strlen(path); - - file->path_original = git__strdup(path); - if (file->path_original == NULL) { - error = GIT_ENOMEM; - goto cleanup; - } - - file->path_lock = git__malloc(path_len + GIT_FILELOCK_EXTLENGTH); - if (file->path_lock == NULL) { - error = GIT_ENOMEM; - goto cleanup; - } - - memcpy(file->path_lock, file->path_original, path_len); - memcpy(file->path_lock + path_len, GIT_FILELOCK_EXTENSION, GIT_FILELOCK_EXTLENGTH); - + /* Allocate the main cache buffer */ file->buffer = git__malloc(file->buf_size); if (file->buffer == NULL){ error = GIT_ENOMEM; goto cleanup; } -#ifdef GIT_FILEBUF_THREADS - file->buffer_back = git__malloc(file->buf_size); - if (file->buffer_back == NULL){ - error = GIT_ENOMEM; - goto cleanup; - } -#endif - + /* If we are hashing on-write, allocate a new hash context */ if (flags & GIT_FILEBUF_HASH_CONTENTS) { if ((file->digest = git_hash_new_ctx()) == NULL) { error = GIT_ENOMEM; @@ -160,8 +175,78 @@ int git_filebuf_open(git_filebuf *file, const char *path, int flags) } } - if ((error = lock_file(file, flags)) < GIT_SUCCESS) - goto cleanup; + /* If we are deflating on-write, */ + if (flags & GIT_FILEBUF_DEFLATE_CONTENTS) { + + /* Initialize the ZLib stream */ + if (deflateInit(&file->zs, Z_DEFAULT_COMPRESSION) != Z_OK) { + error = GIT_EZLIB; + goto cleanup; + } + + /* Allocate the Zlib cache buffer */ + file->z_buf = git__malloc(file->buf_size); + if (file->z_buf == NULL){ + error = GIT_ENOMEM; + goto cleanup; + } + + /* Never flush */ + file->flush_mode = Z_NO_FLUSH; + file->write = &write_deflate; + } else { + file->write = &write_normal; + } + + /* If we are writing to a temp file */ + if (flags & GIT_FILEBUF_TEMPORARY) { + char tmp_path[GIT_PATH_MAX]; + + /* Open the file as temporary for locking */ + file->fd = gitfo_creat_tmp(tmp_path, "_filebuf_"); + if (file->fd < 0) { + error = GIT_EOSERR; + goto cleanup; + } + + /* No original path */ + file->path_original = NULL; + file->path_lock = git__strdup(tmp_path); + + if (file->path_lock == NULL) { + error = GIT_ENOMEM; + goto cleanup; + } + } else { + /* If the file is not temporary, make sure we have a path */ + if (path == NULL) { + error = GIT_ERROR; + goto cleanup; + } + + path_len = strlen(path); + + /* Save the original path of the file */ + file->path_original = git__strdup(path); + if (file->path_original == NULL) { + error = GIT_ENOMEM; + goto cleanup; + } + + /* create the locking path by appending ".lock" to the original */ + file->path_lock = git__malloc(path_len + GIT_FILELOCK_EXTLENGTH); + if (file->path_lock == NULL) { + error = GIT_ENOMEM; + goto cleanup; + } + + memcpy(file->path_lock, file->path_original, path_len); + memcpy(file->path_lock + path_len, GIT_FILELOCK_EXTENSION, GIT_FILELOCK_EXTLENGTH); + + /* open the file for locking */ + if ((error = lock_file(file, flags)) < GIT_SUCCESS) + goto cleanup; + } return GIT_SUCCESS; @@ -187,10 +272,25 @@ int git_filebuf_hash(git_oid *oid, git_filebuf *file) return GIT_SUCCESS; } +int git_filebuf_commit_at(git_filebuf *file, const char *path) +{ + free(file->path_original); + file->path_original = git__strdup(path); + if (file->path_original == NULL) + return GIT_ENOMEM; + + return git_filebuf_commit(file); +} + int git_filebuf_commit(git_filebuf *file) { int error; + /* tmp file cannot be commited */ + if (file->path_original == NULL) + return GIT_EOSERR; + + file->flush_mode = Z_FINISH; if ((error = flush_buffer(file)) < GIT_SUCCESS) goto cleanup; @@ -204,16 +304,16 @@ cleanup: return error; } -GIT_INLINE(void) add_to_cache(git_filebuf *file, void *buf, size_t len) +GIT_INLINE(void) add_to_cache(git_filebuf *file, const void *buf, size_t len) { memcpy(file->buffer + file->buf_pos, buf, len); file->buf_pos += len; } -int git_filebuf_write(git_filebuf *file, void *buff, size_t len) +int git_filebuf_write(git_filebuf *file, const void *buff, size_t len) { int error; - unsigned char *buf = buff; + const unsigned char *buf = buff; for (;;) { size_t space_left = file->buf_size - file->buf_pos; @@ -237,9 +337,9 @@ int git_filebuf_write(git_filebuf *file, void *buff, size_t len) /* write too-large chunks immediately */ if (len > file->buf_size) { - error = gitfo_write(file->fd, buf, len); - if (file->digest) - git_hash_update(file->digest, buf, len); + error = file->write(file, buf, len); + if (error < GIT_SUCCESS) + return error; } } } diff --git a/src/filebuf.h b/src/filebuf.h index 9db615fbd..37cb36784 100644 --- a/src/filebuf.h +++ b/src/filebuf.h @@ -3,14 +3,17 @@ #include "fileops.h" #include "hash.h" +#include "git2/zlib.h" #ifdef GIT_THREADS # define GIT_FILEBUF_THREADS #endif -#define GIT_FILEBUF_HASH_CONTENTS 0x1 -#define GIT_FILEBUF_APPEND 0x2 -#define GIT_FILEBUF_FORCE 0x4 +#define GIT_FILEBUF_HASH_CONTENTS (1 << 0) +#define GIT_FILEBUF_APPEND (1 << 2) +#define GIT_FILEBUF_FORCE (1 << 3) +#define GIT_FILEBUF_TEMPORARY (1 << 4) +#define GIT_FILEBUF_DEFLATE_CONTENTS (1 << 5) #define GIT_FILELOCK_EXTENSION ".lock\0" #define GIT_FILELOCK_EXTLENGTH 6 @@ -19,12 +22,16 @@ struct git_filebuf { char *path_original; char *path_lock; + int (*write)(struct git_filebuf *file, + const void *source, size_t len); + git_hash_ctx *digest; unsigned char *buffer; -#ifdef GIT_FILEBUF_THREADS - unsigned char *buffer_back; -#endif + unsigned char *z_buf; + + z_stream zs; + int flush_mode; size_t buf_size, buf_pos; git_file fd; @@ -32,12 +39,13 @@ struct git_filebuf { typedef struct git_filebuf git_filebuf; -int git_filebuf_write(git_filebuf *lock, void *buff, size_t len); +int git_filebuf_write(git_filebuf *lock, const void *buff, size_t len); int git_filebuf_reserve(git_filebuf *file, void **buff, size_t len); int git_filebuf_printf(git_filebuf *file, const char *format, ...); int git_filebuf_open(git_filebuf *lock, const char *path, int flags); int git_filebuf_commit(git_filebuf *lock); +int git_filebuf_commit_at(git_filebuf *lock, const char *path); void git_filebuf_cleanup(git_filebuf *lock); int git_filebuf_hash(git_oid *oid, git_filebuf *file); diff --git a/src/fileops.c b/src/fileops.c index 76e689e8a..237a16a9f 100644 --- a/src/fileops.c +++ b/src/fileops.c @@ -2,13 +2,13 @@ #include "fileops.h" #include -static int force_path(const char *to) +int gitfo_mkdir_2file(const char *file_path) { const int mode = 0755; /* or 0777 ? */ int error = GIT_SUCCESS; char target_folder_path[GIT_PATH_MAX]; - error = git__dirname_r(target_folder_path, sizeof(target_folder_path), to); + error = git__dirname_r(target_folder_path, sizeof(target_folder_path), file_path); if (error < GIT_SUCCESS) return error; @@ -25,6 +25,87 @@ static int force_path(const char *to) return GIT_SUCCESS; } +static int creat_tempfile(char *path_out, const char *tmp_dir, const char *filename) +{ + int fd; + + git__joinpath(path_out, tmp_dir, filename); + strcat(path_out, "_git2_XXXXXX"); + +#ifdef GIT_WIN32 + /* FIXME: there may be race conditions when multi-threading + * with the library */ + if (_mktemp_s(path_out, GIT_PATH_MAX) != 0) + return GIT_EOSERR; + + fd = gitfo_creat(path_out, 0744); +#else + fd = mkstemp(path_out); +#endif + + return fd >= 0 ? fd : GIT_EOSERR; +} + +static const char *find_tmpdir(void) +{ + static int tmpdir_not_found = 0; + static char temp_dir[GIT_PATH_MAX]; + static const char *env_vars[] = { + "TEMP", "TMP", "TMPDIR" + }; + + unsigned int i, j; + char test_file[GIT_PATH_MAX]; + + if (tmpdir_not_found) + return NULL; + + if (temp_dir[0] != '\0') + return temp_dir; + + for (i = 0; i < ARRAY_SIZE(env_vars); ++i) { + char *env_path; + + env_path = getenv(env_vars[i]); + if (env_path == NULL) + continue; + + strcpy(temp_dir, env_path); + + /* Fix backslashes because Windows environment vars + * are probably fucked up */ + for (j = 0; j < strlen(temp_dir); ++j) + if (temp_dir[j] == '\\') + temp_dir[j] = '/'; + + if (creat_tempfile(test_file, temp_dir, "writetest") >= 0) { + gitfo_unlink(test_file); + return temp_dir; + } + } + + /* last resort: current folder. */ + strcpy(temp_dir, "./"); + if (creat_tempfile(test_file, temp_dir, "writetest") >= 0) { + gitfo_unlink(test_file); + return temp_dir; + } + + tmpdir_not_found = 1; + return NULL; +} + +int gitfo_creat_tmp(char *path_out, const char *filename) +{ + const char *tmp_dir; + + tmp_dir = find_tmpdir(); + if (tmp_dir == NULL) + return GIT_EOSERR; + + return creat_tempfile(path_out, tmp_dir, filename); +} + int gitfo_open(const char *path, int flags) { int fd = open(path, flags | O_BINARY); @@ -39,7 +120,7 @@ int gitfo_creat(const char *path, int mode) int gitfo_creat_force(const char *path, int mode) { - if (force_path(path) < GIT_SUCCESS) + if (gitfo_mkdir_2file(path) < GIT_SUCCESS) return GIT_EOSERR; return gitfo_creat(path, mode); @@ -117,6 +198,7 @@ int gitfo_isdir(const char *path) int gitfo_exists(const char *path) { + assert(path); return access(path, F_OK); } @@ -198,7 +280,7 @@ int gitfo_mv(const char *from, const char *to) int gitfo_mv_force(const char *from, const char *to) { - if (force_path(to) < GIT_SUCCESS) + if (gitfo_mkdir_2file(to) < GIT_SUCCESS) return GIT_EOSERR; return gitfo_mv(from, to); diff --git a/src/fileops.h b/src/fileops.h index 5aa302b54..bc636fc38 100644 --- a/src/fileops.h +++ b/src/fileops.h @@ -58,8 +58,10 @@ extern int gitfo_exists(const char *path); extern int gitfo_open(const char *path, int flags); extern int gitfo_creat(const char *path, int mode); extern int gitfo_creat_force(const char *path, int mode); +extern int gitfo_creat_tmp(char *path_out, const char *filename); extern int gitfo_isdir(const char *path); extern int gitfo_mkdir_recurs(const char *path, int mode); +extern int gitfo_mkdir_2file(const char *path); #define gitfo_close(fd) close(fd) extern int gitfo_read(git_file fd, void *buf, size_t cnt); diff --git a/src/index.c b/src/index.c index 95e56b7d5..6a355e11b 100644 --- a/src/index.c +++ b/src/index.c @@ -324,7 +324,7 @@ int git_index_add(git_index *index, const char *rel_path, int stage) entry.file_size = st.st_size; /* write the blob to disk and get the oid */ - if ((error = git_blob_writefile(&entry.oid, index->repository, full_path)) < GIT_SUCCESS) + if ((error = git_blob_create_fromfile(&entry.oid, index->repository, rel_path)) < GIT_SUCCESS) return error; entry.flags |= (stage << GIT_IDXENTRY_STAGESHIFT); diff --git a/src/object.c b/src/object.c index c432c6da0..0572663eb 100644 --- a/src/object.c +++ b/src/object.c @@ -66,153 +66,6 @@ static struct { { "REF_DELTA", 0, 0 } }; -/* - * Object source methods - * - * Abstract buffer methods that allow the writeback system - * to prepare the contents of any git file in-memory before - * writing them to disk. - */ -static int source_resize(git_odb_source *src) -{ - size_t write_offset, new_size; - void *new_data; - - write_offset = (size_t)((char *)src->write_ptr - (char *)src->raw.data); - - new_size = src->raw.len * 2; - if ((new_data = git__malloc(new_size)) == NULL) - return GIT_ENOMEM; - - memcpy(new_data, src->raw.data, src->written_bytes); - free(src->raw.data); - - src->raw.data = new_data; - src->raw.len = new_size; - src->write_ptr = (char *)new_data + write_offset; - - return GIT_SUCCESS; -} - -int git__source_printf(git_odb_source *source, const char *format, ...) -{ - va_list arglist; - int len; - - assert(source->open && source->write_ptr); - - va_start(arglist, format); - - len = vsnprintf(source->write_ptr, source->raw.len - source->written_bytes, format, arglist); - - while (source->written_bytes + len >= source->raw.len) { - if (source_resize(source) < GIT_SUCCESS) - return GIT_ENOMEM; - - len = vsnprintf(source->write_ptr, source->raw.len - source->written_bytes, format, arglist); - } - - source->write_ptr = (char *)source->write_ptr + len; - source->written_bytes += len; - - return GIT_SUCCESS; -} - -int git__source_write(git_odb_source *source, const void *bytes, size_t len) -{ - assert(source); - - assert(source->open && source->write_ptr); - - while (source->written_bytes + len >= source->raw.len) { - if (source_resize(source) < GIT_SUCCESS) - return GIT_ENOMEM; - } - - memcpy(source->write_ptr, bytes, len); - source->write_ptr = (char *)source->write_ptr + len; - source->written_bytes += len; - - return GIT_SUCCESS; -} - -static void prepare_write(git_object *object) -{ - if (object->source.write_ptr != NULL || object->source.open) - git_object__source_close(object); - - /* TODO: proper size calculation */ - object->source.raw.data = git__malloc(OBJECT_BASE_SIZE); - object->source.raw.len = OBJECT_BASE_SIZE; - - object->source.write_ptr = object->source.raw.data; - object->source.written_bytes = 0; - - object->source.open = 1; -} - -static int write_back(git_object *object) -{ - int error; - git_oid new_id; - - assert(object); - - assert(object->source.open); - assert(object->modified); - - object->source.raw.len = object->source.written_bytes; - - if ((error = git_odb_write(&new_id, object->repo->db, &object->source.raw)) < GIT_SUCCESS) - return error; - - if (object->in_memory) { - int idx = git_vector_search(&object->repo->memory_objects, object); - git_vector_remove(&object->repo->memory_objects, idx); - } else { - git_hashtable_remove(object->repo->objects, &object->id); - } - - git_oid_cpy(&object->id, &new_id); - git_hashtable_insert(object->repo->objects, &object->id, object); - - object->source.write_ptr = NULL; - object->source.written_bytes = 0; - - object->modified = 0; - object->in_memory = 0; - - git_object__source_close(object); - return GIT_SUCCESS; -} - -int git_object__source_open(git_object *object) -{ - int error; - - assert(object && !object->in_memory); - - if (object->source.open) - git_object__source_close(object); - - error = git_odb_read(&object->source.raw, object->repo->db, &object->id); - if (error < GIT_SUCCESS) - return error; - - object->source.open = 1; - return GIT_SUCCESS; -} - -void git_object__source_close(git_object *object) -{ - assert(object); - - if (object->source.open) { - git_rawobj_close(&object->source.raw); - object->source.open = 0; - } -} - static int create_object(git_object **object_out, git_otype type) { git_object *object = NULL; @@ -225,43 +78,19 @@ static int create_object(git_object **object_out, git_otype type) case GIT_OBJ_COMMIT: case GIT_OBJ_TAG: case GIT_OBJ_BLOB: + case GIT_OBJ_TREE: object = git__malloc(git_object__size(type)); if (object == NULL) return GIT_ENOMEM; memset(object, 0x0, git_object__size(type)); break; - - case GIT_OBJ_TREE: - object = (git_object *)git_tree__new(); - if (object == NULL) - return GIT_ENOMEM; - break; default: return GIT_EINVALIDTYPE; } - *object_out = object; - return GIT_SUCCESS; -} + object->type = type; -int git_object_new(git_object **object_out, git_repository *repo, git_otype type) -{ - git_object *object = NULL; - int error; - - assert(object_out && repo); - - if ((error = create_object(&object, type)) < GIT_SUCCESS) - return error; - - object->repo = repo; - object->in_memory = 1; - object->modified = 1; - - object->source.raw.type = type; - - object->lru = ++repo->lru_counter; *object_out = object; return GIT_SUCCESS; } @@ -269,126 +98,77 @@ int git_object_new(git_object **object_out, git_repository *repo, git_otype type int git_object_lookup(git_object **object_out, git_repository *repo, const git_oid *id, git_otype type) { git_object *object = NULL; - git_rawobj obj_file; + git_odb_object *odb_obj; int error = GIT_SUCCESS; assert(repo && object_out && id); - object = git_hashtable_lookup(repo->objects, id); + object = git_cache_get(&repo->objects, id); if (object != NULL) { - if (type != GIT_OBJ_ANY && type != object->source.raw.type) + if (type != GIT_OBJ_ANY && type != object->type) return GIT_EINVALIDTYPE; *object_out = object; - object->lru = ++repo->lru_counter; - object->can_free = 0; return GIT_SUCCESS; } - error = git_odb_read(&obj_file, repo->db, id); + error = git_odb_read(&odb_obj, repo->db, id); if (error < GIT_SUCCESS) return error; - if (type != GIT_OBJ_ANY && type != obj_file.type) { - git_rawobj_close(&obj_file); + if (type != GIT_OBJ_ANY && type != odb_obj->raw.type) { + git_odb_object_close(odb_obj); return GIT_EINVALIDTYPE; } - type = obj_file.type; + type = odb_obj->raw.type; if ((error = create_object(&object, type)) < GIT_SUCCESS) return error; /* Initialize parent object */ - git_oid_cpy(&object->id, id); + git_oid_cpy(&object->cached.oid, id); object->repo = repo; - memcpy(&object->source.raw, &obj_file, sizeof(git_rawobj)); - object->source.open = 1; switch (type) { case GIT_OBJ_COMMIT: - error = git_commit__parse((git_commit *)object); + error = git_commit__parse((git_commit *)object, odb_obj); break; case GIT_OBJ_TREE: - error = git_tree__parse((git_tree *)object); + error = git_tree__parse((git_tree *)object, odb_obj); break; case GIT_OBJ_TAG: - error = git_tag__parse((git_tag *)object); + error = git_tag__parse((git_tag *)object, odb_obj); break; case GIT_OBJ_BLOB: - error = git_blob__parse((git_blob *)object); + error = git_blob__parse((git_blob *)object, odb_obj); break; default: break; } + git_odb_object_close(odb_obj); + if (error < GIT_SUCCESS) { git_object__free(object); return error; } - git_object__source_close(object); - git_hashtable_insert(repo->objects, &object->id, object); - - object->lru = ++repo->lru_counter; - *object_out = object; + *object_out = git_cache_try_store(&repo->objects, object); return GIT_SUCCESS; } -int git_object_write(git_object *object) +void git_object__free(void *_obj) { - int error; - git_odb_source *source; + git_object *object = (git_object *)_obj; assert(object); - if (object->modified == 0) - return GIT_SUCCESS; - - prepare_write(object); - source = &object->source; - - switch (source->raw.type) { - case GIT_OBJ_COMMIT: - error = git_commit__writeback((git_commit *)object, source); - break; - - case GIT_OBJ_TREE: - error = git_tree__writeback((git_tree *)object, source); - break; - - case GIT_OBJ_TAG: - error = git_tag__writeback((git_tag *)object, source); - break; - - case GIT_OBJ_BLOB: - error = git_blob__writeback((git_blob *)object, source); - break; - - default: - error = GIT_ERROR; - break; - } - - if (error < GIT_SUCCESS) { - git_object__source_close(object); - return error; - } - - return write_back(object); -} - -void git_object__free(git_object *object) -{ - assert(object); - - git_object__source_close(object); - - switch (object->source.raw.type) { + switch (object->type) { case GIT_OBJ_COMMIT: git_commit__free((git_commit *)object); break; @@ -416,29 +196,19 @@ void git_object_close(git_object *object) if (object == NULL) return; - if (object->in_memory) { - int idx = git_vector_search(&object->repo->memory_objects, object); - git_vector_remove(&object->repo->memory_objects, idx); - git_object__free(object); - } else { - object->can_free = 1; - } + git_cached_obj_decref((git_cached_obj *)object, git_object__free); } const git_oid *git_object_id(const git_object *obj) { assert(obj); - - if (obj->in_memory) - return NULL; - - return &obj->id; + return &obj->cached.oid; } git_otype git_object_type(const git_object *obj) { assert(obj); - return obj->source.raw.type; + return obj->type; } git_repository *git_object_owner(const git_object *obj) diff --git a/src/odb.c b/src/odb.c index 2013ac24c..9aeaa8a23 100644 --- a/src/odb.c +++ b/src/odb.c @@ -87,55 +87,48 @@ int git_odb__hash_obj(git_oid *id, char *hdr, size_t n, int *len, git_rawobj *ob return GIT_SUCCESS; } -void git_rawobj_close(git_rawobj *obj) + +static git_odb_object *new_odb_object(const git_oid *oid, git_rawobj *source) { - free(obj->data); - obj->data = NULL; + git_odb_object *object = git__malloc(sizeof(git_odb_object)); + memset(object, 0x0, sizeof(git_odb_object)); + + git_oid_cpy(&object->cached.oid, oid); + memcpy(&object->raw, source, sizeof(git_rawobj)); + + return object; } -int git_rawobj_hash(git_oid *id, git_rawobj *obj) +static void free_odb_object(void *o) +{ + git_odb_object *object = (git_odb_object *)o; + + if (object != NULL) { + free(object->raw.data); + free(object); + } +} + +void git_odb_object_close(git_odb_object *object) +{ + git_cached_obj_decref((git_cached_obj *)object, &free_odb_object); +} + +int git_odb_hash(git_oid *id, const void *data, size_t len, git_otype type) { char hdr[64]; int hdrlen; + git_rawobj raw; - assert(id && obj); + assert(id); - return git_odb__hash_obj(id, hdr, sizeof(hdr), &hdrlen, obj); + raw.data = (void *)data; + raw.len = len; + raw.type = type; + + return git_odb__hash_obj(id, hdr, sizeof(hdr), &hdrlen, &raw); } -int git_odb__inflate_buffer(void *in, size_t inlen, void *out, size_t outlen) -{ - z_stream zs; - int status = Z_OK; - - memset(&zs, 0x0, sizeof(zs)); - - zs.next_out = out; - zs.avail_out = outlen; - - zs.next_in = in; - zs.avail_in = inlen; - - if (inflateInit(&zs) < Z_OK) - return GIT_ERROR; - - while (status == Z_OK) - status = inflate(&zs, Z_FINISH); - - inflateEnd(&zs); - - if ((status != Z_STREAM_END) /*|| (zs.avail_in != 0) */) - return GIT_ERROR; - - if (zs.total_out != outlen) - return GIT_ERROR; - - return GIT_SUCCESS; -} - - - - /*********************************************************** * @@ -162,6 +155,8 @@ int git_odb_new(git_odb **out) if (!db) return GIT_ENOMEM; + git_cache_init(&db->cache, GIT_DEFAULT_CACHE_SIZE, &free_odb_object); + if (git_vector_init(&db->backends, 4, backend_sort_cmp) < 0) { free(db); return GIT_ENOMEM; @@ -306,16 +301,23 @@ void git_odb_close(git_odb *db) } git_vector_free(&db->backends); + git_cache_free(&db->cache); free(db); } int git_odb_exists(git_odb *db, const git_oid *id) { + git_odb_object *object; unsigned int i; int found = 0; assert(db && id); + if ((object = git_cache_get(&db->cache, id)) != NULL) { + git_odb_object_close(object); + return 1; + } + for (i = 0; i < db->backends.length && !found; ++i) { backend_internal *internal = git_vector_get(&db->backends, i); git_odb_backend *b = internal->backend; @@ -327,19 +329,27 @@ int git_odb_exists(git_odb *db, const git_oid *id) return found; } -int git_odb_read_header(git_rawobj *out, git_odb *db, const git_oid *id) +int git_odb_read_header(size_t *len_p, git_otype *type_p, git_odb *db, const git_oid *id) { unsigned int i; int error = GIT_ENOTFOUND; + git_odb_object *object; - assert(out && db && id); + assert(db && id); + + if ((object = git_cache_get(&db->cache, id)) != NULL) { + *len_p = object->raw.len; + *type_p = object->raw.type; + git_odb_object_close(object); + return GIT_SUCCESS; + } for (i = 0; i < db->backends.length && error < 0; ++i) { backend_internal *internal = git_vector_get(&db->backends, i); git_odb_backend *b = internal->backend; if (b->read_header != NULL) - error = b->read_header(out, b, id); + error = b->read_header(len_p, type_p, b, id); } /* @@ -347,37 +357,50 @@ int git_odb_read_header(git_rawobj *out, git_odb *db, const git_oid *id) * try reading the whole object and freeing the contents */ if (error < 0) { - error = git_odb_read(out, db, id); - git_rawobj_close(out); + if ((error = git_odb_read(&object, db, id)) < GIT_SUCCESS) + return error; + + *len_p = object->raw.len; + *type_p = object->raw.len; + git_odb_object_close(object); } - return error; + return GIT_SUCCESS; } -int git_odb_read(git_rawobj *out, git_odb *db, const git_oid *id) +int git_odb_read(git_odb_object **out, git_odb *db, const git_oid *id) { unsigned int i; int error = GIT_ENOTFOUND; + git_rawobj raw; assert(out && db && id); + *out = git_cache_get(&db->cache, id); + if (*out != NULL) + return GIT_SUCCESS; + for (i = 0; i < db->backends.length && error < 0; ++i) { backend_internal *internal = git_vector_get(&db->backends, i); git_odb_backend *b = internal->backend; if (b->read != NULL) - error = b->read(out, b, id); + error = b->read(&raw.data, &raw.len, &raw.type, b, id); + } + + if (error == GIT_SUCCESS) { + *out = git_cache_try_store(&db->cache, new_odb_object(id, &raw)); } return error; } -int git_odb_write(git_oid *id, git_odb *db, git_rawobj *obj) +int git_odb_open_wstream(git_odb_stream **stream, git_odb *db, size_t size, git_otype type) { unsigned int i; int error = GIT_ERROR; - assert(obj && db && id); + assert(stream && db); for (i = 0; i < db->backends.length && error < 0; ++i) { backend_internal *internal = git_vector_get(&db->backends, i); @@ -387,8 +410,26 @@ int git_odb_write(git_oid *id, git_odb *db, git_rawobj *obj) if (internal->is_alternate) continue; - if (b->write != NULL) - error = b->write(id, b, obj); + if (b->writestream != NULL) + error = b->writestream(stream, b, size, type); + } + + return error; +} + +int git_odb_open_rstream(git_odb_stream **stream, git_odb *db, const git_oid *oid) +{ + unsigned int i; + int error = GIT_ERROR; + + assert(stream && db); + + for (i = 0; i < db->backends.length && error < 0; ++i) { + backend_internal *internal = git_vector_get(&db->backends, i); + git_odb_backend *b = internal->backend; + + if (b->readstream != NULL) + error = b->readstream(stream, b, oid); } return error; diff --git a/src/odb.h b/src/odb.h index c3d0a17ab..f3685834e 100644 --- a/src/odb.h +++ b/src/odb.h @@ -3,15 +3,31 @@ #include "git2/odb.h" #include "git2/oid.h" +#include "git2/types.h" #include "vector.h" +#include "cache.h" +/* DO NOT EXPORT */ +typedef struct { + void *data; /**< Raw, decompressed object data. */ + size_t len; /**< Total number of bytes in data. */ + git_otype type; /**< Type of this object. */ +} git_rawobj; + +/* EXPORT */ +struct git_odb_object { + git_cached_obj cached; + git_rawobj raw; +}; + +/* EXPORT */ struct git_odb { void *_internal; git_vector backends; + git_cache cache; }; int git_odb__hash_obj(git_oid *id, char *hdr, size_t n, int *len, git_rawobj *obj); -int git_odb__inflate_buffer(void *in, size_t inlen, void *out, size_t outlen); #endif diff --git a/src/odb_loose.c b/src/odb_loose.c index 4e2d9a639..4ab1128f3 100644 --- a/src/odb_loose.c +++ b/src/odb_loose.c @@ -30,14 +30,22 @@ #include "hash.h" #include "odb.h" #include "delta-apply.h" +#include "filebuf.h" #include "git2/odb_backend.h" +#include "git2/types.h" typedef struct { /* object header data */ git_otype type; /* object type */ size_t size; /* object size */ } obj_hdr; +typedef struct { + git_odb_stream stream; + git_filebuf fbuf; + int finished; +} loose_writestream; + typedef struct loose_backend { git_odb_backend parent; @@ -53,38 +61,6 @@ typedef struct loose_backend { * ***********************************************************/ -static int make_temp_file(git_file *fd, char *tmp, size_t n, char *file) -{ - char *template = "/tmp_obj_XXXXXX"; - size_t tmplen = strlen(template); - int dirlen; - - if ((dirlen = git__dirname_r(tmp, n, file)) < 0) - return GIT_ERROR; - - if ((dirlen + tmplen) >= n) - return GIT_ERROR; - - strcpy(tmp + dirlen, (dirlen) ? template : template + 1); - - *fd = gitfo_mkstemp(tmp); - if (*fd < 0 && dirlen) { - /* create directory if it doesn't exist */ - tmp[dirlen] = '\0'; - if ((gitfo_exists(tmp) < 0) && gitfo_mkdir(tmp, 0755)) - return GIT_ERROR; - /* try again */ - strcpy(tmp + dirlen, template); - *fd = gitfo_mkstemp(tmp); - } - if (*fd < 0) - return GIT_ERROR; - - return GIT_SUCCESS; -} - - - static size_t object_file_name(char *name, size_t n, char *dir, const git_oid *id) { size_t len = strlen(dir); @@ -236,64 +212,6 @@ static int finish_inflate(z_stream *s) return GIT_SUCCESS; } -static int deflate_buf(z_stream *s, void *in, size_t len, int flush) -{ - int status = Z_OK; - - set_stream_input(s, in, len); - while (status == Z_OK) { - status = deflate(s, flush); - if (s->avail_in == 0) - break; - } - return status; -} - -static int deflate_obj(gitfo_buf *buf, char *hdr, int hdrlen, git_rawobj *obj, int level) -{ - z_stream zs; - int status; - size_t size; - - assert(buf && !buf->data && hdr && obj); - assert(level == Z_DEFAULT_COMPRESSION || (level >= 0 && level <= 9)); - - buf->data = NULL; - buf->len = 0; - init_stream(&zs, NULL, 0); - - if (deflateInit(&zs, level) < Z_OK) - return GIT_ERROR; - - size = deflateBound(&zs, hdrlen + obj->len); - - if ((buf->data = git__malloc(size)) == NULL) { - deflateEnd(&zs); - return GIT_ERROR; - } - - set_stream_output(&zs, buf->data, size); - - /* compress the header */ - status = deflate_buf(&zs, hdr, hdrlen, Z_NO_FLUSH); - - /* if header compressed OK, compress the object */ - if (status == Z_OK) - status = deflate_buf(&zs, obj->data, obj->len, Z_FINISH); - - if (status != Z_STREAM_END) { - deflateEnd(&zs); - free(buf->data); - buf->data = NULL; - return GIT_ERROR; - } - - buf->len = zs.total_out; - deflateEnd(&zs); - - return GIT_SUCCESS; -} - static int is_zlib_compressed_data(unsigned char *data) { unsigned int w; @@ -302,6 +220,36 @@ static int is_zlib_compressed_data(unsigned char *data) return data[0] == 0x78 && !(w % 31); } +static int inflate_buffer(void *in, size_t inlen, void *out, size_t outlen) +{ + z_stream zs; + int status = Z_OK; + + memset(&zs, 0x0, sizeof(zs)); + + zs.next_out = out; + zs.avail_out = outlen; + + zs.next_in = in; + zs.avail_in = inlen; + + if (inflateInit(&zs) < Z_OK) + return GIT_ERROR; + + while (status == Z_OK) + status = inflate(&zs, Z_FINISH); + + inflateEnd(&zs); + + if ((status != Z_STREAM_END) /*|| (zs.avail_in != 0) */) + return GIT_ERROR; + + if (zs.total_out != outlen) + return GIT_ERROR; + + return GIT_SUCCESS; +} + static void *inflate_tail(z_stream *s, void *hb, size_t used, obj_hdr *hdr) { unsigned char *buf, *head = hb; @@ -371,7 +319,7 @@ static int inflate_packlike_loose_disk_obj(git_rawobj *out, gitfo_buf *obj) in = ((unsigned char *)obj->data) + used; len = obj->len - used; - if (git_odb__inflate_buffer(in, len, buf, hdr.size)) { + if (inflate_buffer(in, len, buf, hdr.size)) { free(buf); return GIT_ERROR; } @@ -505,37 +453,6 @@ cleanup: return error; } -static int write_obj(gitfo_buf *buf, git_oid *id, loose_backend *backend) -{ - char file[GIT_PATH_MAX]; - char temp[GIT_PATH_MAX]; - git_file fd; - - if (object_file_name(file, sizeof(file), backend->objects_dir, id)) - return GIT_EOSERR; - - if (make_temp_file(&fd, temp, sizeof(temp), file) < 0) - return GIT_EOSERR; - - if (gitfo_write(fd, buf->data, buf->len) < 0) { - gitfo_close(fd); - gitfo_unlink(temp); - return GIT_EOSERR; - } - - if (backend->fsync_object_files) - gitfo_fsync(fd); - gitfo_close(fd); - gitfo_chmod(temp, 0444); - - if (gitfo_mv(temp, file) < 0) { - gitfo_unlink(temp); - return GIT_EOSERR; - } - - return GIT_SUCCESS; -} - static int locate_object(char *object_location, loose_backend *backend, const git_oid *oid) { object_file_name(object_location, GIT_PATH_MAX, backend->objects_dir, oid); @@ -558,29 +475,44 @@ static int locate_object(char *object_location, loose_backend *backend, const gi * ***********************************************************/ -int loose_backend__read_header(git_rawobj *obj, git_odb_backend *backend, const git_oid *oid) +int loose_backend__read_header(size_t *len_p, git_otype *type_p, git_odb_backend *backend, const git_oid *oid) { char object_path[GIT_PATH_MAX]; + git_rawobj raw; + int error; - assert(obj && backend && oid); + assert(backend && oid); if (locate_object(object_path, (loose_backend *)backend, oid) < 0) return GIT_ENOTFOUND; - return read_header_loose(obj, object_path); + if ((error = read_header_loose(&raw, object_path)) < GIT_SUCCESS) + return error; + + *len_p = raw.len; + *type_p = raw.type; + return GIT_SUCCESS; } - -int loose_backend__read(git_rawobj *obj, git_odb_backend *backend, const git_oid *oid) +int loose_backend__read(void **buffer_p, size_t *len_p, git_otype *type_p, git_odb_backend *backend, const git_oid *oid) { char object_path[GIT_PATH_MAX]; + git_rawobj raw; + int error; - assert(obj && backend && oid); + assert(backend && oid); if (locate_object(object_path, (loose_backend *)backend, oid) < 0) return GIT_ENOTFOUND; - return read_loose(obj, object_path); + if ((error = read_loose(&raw, object_path)) < GIT_SUCCESS) + return error; + + *buffer_p = raw.data; + *len_p = raw.len; + *type_p = raw.type; + + return GIT_SUCCESS; } int loose_backend__exists(git_odb_backend *backend, const git_oid *oid) @@ -592,32 +524,104 @@ int loose_backend__exists(git_odb_backend *backend, const git_oid *oid) return locate_object(object_path, (loose_backend *)backend, oid) == GIT_SUCCESS; } - -int loose_backend__write(git_oid *id, git_odb_backend *_backend, git_rawobj *obj) +int loose_backend__stream_fwrite(git_oid *oid, git_odb_stream *_stream) { + loose_writestream *stream = (loose_writestream *)_stream; + loose_backend *backend = (loose_backend *)_stream->backend; + + int error; + char final_path[GIT_PATH_MAX]; + + if ((error = git_filebuf_hash(oid, &stream->fbuf)) < GIT_SUCCESS) + return error; + + if (object_file_name(final_path, sizeof(final_path), backend->objects_dir, oid)) + return GIT_ENOMEM; + + if ((error = gitfo_mkdir_2file(final_path)) < GIT_SUCCESS) + return error; + + stream->finished = 1; + return git_filebuf_commit_at(&stream->fbuf, final_path); +} + +int loose_backend__stream_write(git_odb_stream *_stream, const char *data, size_t len) +{ + loose_writestream *stream = (loose_writestream *)_stream; + return git_filebuf_write(&stream->fbuf, data, len); +} + +void loose_backend__stream_free(git_odb_stream *_stream) +{ + loose_writestream *stream = (loose_writestream *)_stream; + + if (!stream->finished) + git_filebuf_cleanup(&stream->fbuf); + + free(stream); +} + +static int format_object_header(char *hdr, size_t n, size_t obj_len, git_otype obj_type) +{ + const char *type_str = git_object_type2string(obj_type); + int len = snprintf(hdr, n, "%s %"PRIuZ, type_str, obj_len); + + assert(len > 0); /* otherwise snprintf() is broken */ + assert(((size_t) len) < n); /* otherwise the caller is broken! */ + + if (len < 0 || ((size_t) len) >= n) + return GIT_ERROR; + return len+1; +} + +int loose_backend__stream(git_odb_stream **stream_out, git_odb_backend *_backend, size_t length, git_otype type) +{ + loose_backend *backend; + loose_writestream *stream; + char hdr[64]; int hdrlen; - gitfo_buf buf = GITFO_BUF_INIT; int error; - loose_backend *backend; - assert(id && _backend && obj); + assert(_backend); backend = (loose_backend *)_backend; + *stream_out = NULL; - if ((error = git_odb__hash_obj(id, hdr, sizeof(hdr), &hdrlen, obj)) < 0) + hdrlen = format_object_header(hdr, sizeof(hdr), length, type); + if (hdrlen < GIT_SUCCESS) + return GIT_EOBJCORRUPTED; + + stream = git__calloc(1, sizeof(loose_writestream)); + if (stream == NULL) + return GIT_ENOMEM; + + stream->stream.backend = _backend; + stream->stream.read = NULL; /* read only */ + stream->stream.write = &loose_backend__stream_write; + stream->stream.finalize_write = &loose_backend__stream_fwrite; + stream->stream.free = &loose_backend__stream_free; + stream->stream.mode = GIT_STREAM_WRONLY; + + error = git_filebuf_open(&stream->fbuf, NULL, + GIT_FILEBUF_HASH_CONTENTS | + GIT_FILEBUF_DEFLATE_CONTENTS | + GIT_FILEBUF_TEMPORARY); + + if (error < GIT_SUCCESS) { + free(stream); return error; + } - if (git_odb_exists(_backend->odb, id)) - return GIT_SUCCESS; - - if ((error = deflate_obj(&buf, hdr, hdrlen, obj, backend->object_zlib_level)) < 0) + error = stream->stream.write((git_odb_stream *)stream, hdr, hdrlen); + if (error < GIT_SUCCESS) { + git_filebuf_cleanup(&stream->fbuf); + free(stream); return error; + } - error = write_obj(&buf, id, backend); - - gitfo_free_buf(&buf); - return error; + *stream_out = (git_odb_stream *)stream; + return GIT_SUCCESS; } void loose_backend__free(git_odb_backend *_backend) @@ -649,7 +653,7 @@ int git_odb_backend_loose(git_odb_backend **backend_out, const char *objects_dir backend->parent.read = &loose_backend__read; backend->parent.read_header = &loose_backend__read_header; - backend->parent.write = &loose_backend__write; + backend->parent.writestream = &loose_backend__stream; backend->parent.exists = &loose_backend__exists; backend->parent.free = &loose_backend__free; diff --git a/src/odb_pack.c b/src/odb_pack.c index 3067179be..65210f0b0 100644 --- a/src/odb_pack.c +++ b/src/odb_pack.c @@ -1243,7 +1243,7 @@ static int packfile_unpack_delta( error = packfile_unpack_compressed(&delta, backend, p, w_curs, curpos, delta_size, delta_type); if (error < GIT_SUCCESS) { - git_rawobj_close(&base); + free(base.data); return error; } @@ -1252,8 +1252,8 @@ static int packfile_unpack_delta( base.data, base.len, delta.data, delta.len); - git_rawobj_close(&base); - git_rawobj_close(&delta); + free(base.data); + free(delta.data); /* TODO: we might want to cache this shit. eventually */ //add_delta_base_cache(p, base_offset, base, base_size, *type); @@ -1337,15 +1337,23 @@ int pack_backend__read_header(git_rawobj *obj, git_odb_backend *backend, const g } */ -int pack_backend__read(git_rawobj *obj, git_odb_backend *backend, const git_oid *oid) +int pack_backend__read(void **buffer_p, size_t *len_p, git_otype *type_p, git_odb_backend *backend, const git_oid *oid) { struct pack_entry e; + git_rawobj raw; int error; if ((error = pack_entry_find(&e, (struct pack_backend *)backend, oid)) < GIT_SUCCESS) return error; - return packfile_unpack(obj, (struct pack_backend *)backend, e.p, e.offset); + if ((error = packfile_unpack(&raw, (struct pack_backend *)backend, e.p, e.offset)) < GIT_SUCCESS) + return error; + + *buffer_p = raw.data; + *len_p = raw.len; + *type_p = raw.type; + + return GIT_SUCCESS; } int pack_backend__exists(git_odb_backend *backend, const git_oid *oid) @@ -1397,7 +1405,6 @@ int git_odb_backend_pack(git_odb_backend **backend_out, const char *objects_dir) backend->parent.read = &pack_backend__read; backend->parent.read_header = NULL; - backend->parent.write = NULL; backend->parent.exists = &pack_backend__exists; backend->parent.free = &pack_backend__free; diff --git a/src/oid.c b/src/oid.c index 81b7d6005..eb167a685 100644 --- a/src/oid.c +++ b/src/oid.c @@ -143,14 +143,18 @@ int git__parse_oid(git_oid *oid, char **buffer_out, return GIT_SUCCESS; } -int git__write_oid(git_odb_source *src, const char *header, const git_oid *oid) +int git__write_oid(git_odb_stream *stream, const char *header, const git_oid *oid) { - char hex_oid[41]; + char hex_oid[42]; - git_oid_fmt(hex_oid, oid); - hex_oid[40] = 0; + git_oid_fmt(hex_oid + 1, oid); - return git__source_printf(src, "%s %s\n", header, hex_oid); + hex_oid[0] = ' '; + hex_oid[41] = '\n'; + + stream->write(stream, header, strlen(header)); + stream->write(stream, hex_oid, 42); + return GIT_SUCCESS; } void git_oid_mkraw(git_oid *out, const unsigned char *raw) diff --git a/src/repository.c b/src/repository.c index 37aa44781..132969402 100644 --- a/src/repository.c +++ b/src/repository.c @@ -40,29 +40,11 @@ #define GIT_BRANCH_MASTER "master" -static const int OBJECT_TABLE_SIZE = 32; - typedef struct { char *path_repository; unsigned is_bare:1, has_been_reinit:1; } repo_init; -/* - * Hash table methods - * - * Callbacks for the ODB cache, implemented - * as a hash table - */ -static uint32_t object_table_hash(const void *key, int hash_id) -{ - uint32_t r; - git_oid *id; - - id = (git_oid *)key; - memcpy(&r, id->id + (hash_id * sizeof(uint32_t)), sizeof(r)); - return r; -} - /* * Git repository open methods * @@ -186,25 +168,9 @@ static git_repository *repository_alloc() memset(repo, 0x0, sizeof(git_repository)); - repo->objects = git_hashtable_alloc( - OBJECT_TABLE_SIZE, - object_table_hash, - (git_hash_keyeq_ptr)git_oid_cmp); - - if (repo->objects == NULL) { - free(repo); - return NULL; - } + git_cache_init(&repo->objects, GIT_DEFAULT_CACHE_SIZE, &git_object__free); if (git_repository__refcache_init(&repo->references) < GIT_SUCCESS) { - git_hashtable_free(repo->objects); - free(repo); - return NULL; - } - - if (git_vector_init(&repo->memory_objects, 16, NULL) < GIT_SUCCESS) { - git_hashtable_free(repo->objects); - git_repository__refcache_free(&repo->references); free(repo); return NULL; } @@ -330,51 +296,19 @@ cleanup: return error; } -int git_repository_gc(git_repository *repo) -{ - int collected = 0; - git_object *object; - const void *_unused; - - GIT_HASHTABLE_FOREACH(repo->objects, _unused, object, - if (object->can_free) { - git_object__free(object); - collected++; - } - ); - - return collected; -} - void git_repository_free(git_repository *repo) { - git_object *object; - const void *_unused; - unsigned int i; - if (repo == NULL) return; - /* force free all the objects */ - GIT_HASHTABLE_FOREACH(repo->objects, _unused, object, - git_object__free(object); - ); - - for (i = 0; i < repo->memory_objects.length; ++i) { - object = git_vector_get(&repo->memory_objects, i); - git_object__free(object); - } + git_cache_free(&repo->objects); + git_repository__refcache_free(&repo->references); free(repo->path_workdir); free(repo->path_index); free(repo->path_repository); free(repo->path_odb); - git_hashtable_free(repo->objects); - git_vector_free(&repo->memory_objects); - - git_repository__refcache_free(&repo->references); - if (repo->db != NULL) git_odb_close(repo->db); diff --git a/src/repository.h b/src/repository.h index 48b8dae6b..fef1c7da0 100644 --- a/src/repository.h +++ b/src/repository.h @@ -9,6 +9,7 @@ #include "hashtable.h" #include "index.h" +#include "cache.h" #include "refs.h" #define DOT_GIT ".git" @@ -16,28 +17,17 @@ #define GIT_OBJECTS_DIR "objects/" #define GIT_INDEX_FILE "index" -typedef struct { - git_rawobj raw; - void *write_ptr; - size_t written_bytes; - int open:1; -} git_odb_source; - struct git_object { - git_oid id; + git_cached_obj cached; git_repository *repo; - git_odb_source source; - unsigned int lru; - unsigned char in_memory, modified, can_free, _pad; + git_otype type; }; struct git_repository { git_odb *db; git_index *index; - git_hashtable *objects; - git_vector memory_objects; - + git_cache objects; git_refcache references; char *path_repository; @@ -49,17 +39,11 @@ struct git_repository { unsigned int lru_counter; }; -int git_object__source_open(git_object *object); -void git_object__source_close(git_object *object); - /* fully free the object; internal method, do not * export */ -void git_object__free(git_object *object); - -int git__source_printf(git_odb_source *source, const char *format, ...); -int git__source_write(git_odb_source *source, const void *bytes, size_t len); +void git_object__free(void *object); int git__parse_oid(git_oid *oid, char **buffer_out, const char *buffer_end, const char *header); -int git__write_oid(git_odb_source *src, const char *header, const git_oid *oid); +int git__write_oid(git_odb_stream *src, const char *header, const git_oid *oid); #endif diff --git a/src/revwalk.c b/src/revwalk.c index 3b7ad34e6..a9d4f8734 100644 --- a/src/revwalk.c +++ b/src/revwalk.c @@ -25,6 +25,7 @@ #include "common.h" #include "commit.h" +#include "odb.h" #include "hashtable.h" #include "pqueue.h" @@ -236,22 +237,22 @@ static int commit_quick_parse(git_revwalk *walk, commit_object *commit, git_rawo static int commit_parse(git_revwalk *walk, commit_object *commit) { - git_rawobj data; + git_odb_object *obj; int error; if (commit->parsed) return GIT_SUCCESS; - if ((error = git_odb_read(&data, walk->repo->db, &commit->oid)) < GIT_SUCCESS) + if ((error = git_odb_read(&obj, walk->repo->db, &commit->oid)) < GIT_SUCCESS) return error; - if (data.type != GIT_OBJ_COMMIT) { - git_rawobj_close(&data); + if (obj->raw.type != GIT_OBJ_COMMIT) { + git_odb_object_close(obj); return GIT_EOBJTYPE; } - error = commit_quick_parse(walk, commit, &data); - git_rawobj_close(&data); + error = commit_quick_parse(walk, commit, &obj->raw); + git_odb_object_close(obj); return error; } diff --git a/src/signature.c b/src/signature.c index 5c9f15973..13816c396 100644 --- a/src/signature.c +++ b/src/signature.c @@ -46,13 +46,7 @@ git_signature *git_signature_new(const char *name, const char *email, time_t tim goto cleanup; p->name = git__strdup(name); - if (p->name == NULL) - goto cleanup; - p->email = git__strdup(email); - if (p->email == NULL) - goto cleanup; - p->when.time = time; p->when.offset = offset; @@ -179,10 +173,12 @@ int git_signature__parse(git_signature *sig, char **buffer_out, return GIT_SUCCESS; } -int git_signature__write(git_odb_source *src, const char *header, const git_signature *sig) +int git_signature__write(char **signature, const char *header, const git_signature *sig) { - char sign; int offset, hours, mins; + char sig_buffer[2048]; + int sig_buffer_len; + char sign; offset = sig->when.offset; sign = (sig->when.offset < 0) ? '-' : '+'; @@ -193,7 +189,16 @@ int git_signature__write(git_odb_source *src, const char *header, const git_sign hours = offset / 60; mins = offset % 60; - return git__source_printf(src, "%s %s <%s> %u %c%02d%02d\n", header, sig->name, sig->email, (unsigned)sig->when.time, sign, hours, mins); + sig_buffer_len = snprintf(sig_buffer, sizeof(sig_buffer), + "%s %s <%s> %u %c%02d%02d\n", + header, sig->name, sig->email, + (unsigned)sig->when.time, sign, hours, mins); + + if (sig_buffer_len < 0 || (size_t)sig_buffer_len > sizeof(sig_buffer)) + return GIT_ENOMEM; + + *signature = git__strdup(sig_buffer); + return sig_buffer_len; } diff --git a/src/signature.h b/src/signature.h index ee212c2dc..3534cb21f 100644 --- a/src/signature.h +++ b/src/signature.h @@ -7,6 +7,6 @@ #include int git_signature__parse(git_signature *sig, char **buffer_out, const char *buffer_end, const char *header); -int git_signature__write(git_odb_source *src, const char *header, const git_signature *sig); +int git_signature__write(char **signature, const char *header, const git_signature *sig); #endif diff --git a/src/tag.c b/src/tag.c index 0489e7e4a..7baababbf 100644 --- a/src/tag.c +++ b/src/tag.c @@ -56,21 +56,6 @@ const git_oid *git_tag_target_oid(git_tag *t) return &t->target; } -int git_tag_set_target(git_tag *tag, git_object *target) -{ - const git_oid *oid; - - assert(tag && target); - - if ((oid = git_object_id(target)) == NULL) - return GIT_EMISSINGOBJDATA; - - tag->object.modified = 1; - git_oid_cpy(&tag->target, oid); - tag->type = git_object_type(target); - return GIT_SUCCESS; -} - git_otype git_tag_type(git_tag *t) { assert(t); @@ -83,50 +68,17 @@ const char *git_tag_name(git_tag *t) return t->tag_name; } -void git_tag_set_name(git_tag *tag, const char *name) -{ - assert(tag && name); - - tag->object.modified = 1; - - if (tag->tag_name) - free(tag->tag_name); - - tag->tag_name = git__strdup(name); -} - const git_signature *git_tag_tagger(git_tag *t) { return t->tagger; } -void git_tag_set_tagger(git_tag *tag, const git_signature *tagger_sig) -{ - assert(tag && tagger_sig); - tag->object.modified = 1; - - git_signature_free(tag->tagger); - tag->tagger = git_signature_dup(tagger_sig); -} - const char *git_tag_message(git_tag *t) { assert(t); return t->message; } -void git_tag_set_message(git_tag *tag, const char *message) -{ - assert(tag && message); - - tag->object.modified = 1; - - if (tag->message) - free(tag->message); - - tag->message = git__strdup(message); -} - static int parse_tag_buffer(git_tag *tag, char *buffer, const char *buffer_end) { static const char *tag_types[] = { @@ -187,9 +139,6 @@ static int parse_tag_buffer(git_tag *tag, char *buffer, const char *buffer_end) buffer = search + 1; - if (tag->tagger != NULL) - git_signature_free(tag->tagger); - tag->tagger = git__malloc(sizeof(git_signature)); if ((error = git_signature__parse(tag->tagger, &buffer, buffer_end, "tagger ")) != 0) @@ -197,9 +146,6 @@ static int parse_tag_buffer(git_tag *tag, char *buffer, const char *buffer_end) text_len = buffer_end - ++buffer; - if (tag->message != NULL) - free(tag->message); - tag->message = git__malloc(text_len + 1); memcpy(tag->message, buffer, text_len); tag->message[text_len] = '\0'; @@ -207,26 +153,90 @@ static int parse_tag_buffer(git_tag *tag, char *buffer, const char *buffer_end) return GIT_SUCCESS; } -int git_tag__writeback(git_tag *tag, git_odb_source *src) +int git_tag_create_o( + git_oid *oid, + git_repository *repo, + const char *tag_name, + const git_object *target, + const git_signature *tagger, + const char *message) { - if (tag->tag_name == NULL || tag->tagger == NULL) - return GIT_EMISSINGOBJDATA; + return git_tag_create( + oid, repo, tag_name, + git_object_id(target), + git_object_type(target), + tagger, message); +} - git__write_oid(src, "object", &tag->target); - git__source_printf(src, "type %s\n", git_object_type2string(tag->type)); - git__source_printf(src, "tag %s\n", tag->tag_name); - git_signature__write(src, "tagger", tag->tagger); +int git_tag_create( + git_oid *oid, + git_repository *repo, + const char *tag_name, + const git_oid *target, + git_otype target_type, + const git_signature *tagger, + const char *message) +{ + size_t final_size = 0; + git_odb_stream *stream; - if (tag->message != NULL) - git__source_printf(src, "\n%s", tag->message); + const char *type_str; + char *tagger_str; - return GIT_SUCCESS; + int type_str_len, tag_name_len, tagger_str_len, message_len; + int error; + + + type_str = git_object_type2string(target_type); + + tagger_str_len = git_signature__write(&tagger_str, "tagger", tagger); + + type_str_len = strlen(type_str); + tag_name_len = strlen(tag_name); + message_len = strlen(message); + + final_size += GIT_OID_LINE_LENGTH("object"); + final_size += STRLEN("type ") + type_str_len + 1; + final_size += STRLEN("tag ") + tag_name_len + 1; + final_size += tagger_str_len; + final_size += 1 + message_len; + + if ((error = git_odb_open_wstream(&stream, repo->db, final_size, GIT_OBJ_TAG)) < GIT_SUCCESS) + return error; + + git__write_oid(stream, "object", target); + + stream->write(stream, "type ", STRLEN("type ")); + stream->write(stream, type_str, type_str_len); + + stream->write(stream, "\ntag ", STRLEN("\ntag ")); + stream->write(stream, tag_name, tag_name_len); + stream->write(stream, "\n", 1); + + stream->write(stream, tagger_str, tagger_str_len); + free(tagger_str); + + stream->write(stream, "\n", 1); + stream->write(stream, message, message_len); + + + error = stream->finalize_write(oid, stream); + stream->free(stream); + + if (error == GIT_SUCCESS) { + char ref_name[512]; + git_reference *new_ref; + git__joinpath(ref_name, GIT_REFS_TAGS_DIR, tag_name); + error = git_reference_create_oid(&new_ref, repo, ref_name, oid); + } + + return error; } -int git_tag__parse(git_tag *tag) +int git_tag__parse(git_tag *tag, git_odb_object *obj) { - assert(tag && tag->object.source.open); - return parse_tag_buffer(tag, tag->object.source.raw.data, (char *)tag->object.source.raw.data + tag->object.source.raw.len); + assert(tag); + return parse_tag_buffer(tag, obj->raw.data, (char *)obj->raw.data + obj->raw.len); } diff --git a/src/tag.h b/src/tag.h index a1782d064..eddf8fa3a 100644 --- a/src/tag.h +++ b/src/tag.h @@ -3,6 +3,7 @@ #include "git2/tag.h" #include "repository.h" +#include "odb.h" struct git_tag { git_object object; @@ -16,7 +17,6 @@ struct git_tag { }; void git_tag__free(git_tag *tag); -int git_tag__parse(git_tag *tag); -int git_tag__writeback(git_tag *tag, git_odb_source *src); +int git_tag__parse(git_tag *tag, git_odb_object *obj); #endif diff --git a/src/thread-utils.h b/src/thread-utils.h index 1cf0e3407..e542639c8 100644 --- a/src/thread-utils.h +++ b/src/thread-utils.h @@ -8,7 +8,7 @@ typedef struct { volatile int val; } git_atomic; -static inline void git_atomic_set(git_atomic *a, int val) +GIT_INLINE(void) git_atomic_set(git_atomic *a, int val) { a->val = val; } @@ -36,7 +36,7 @@ static inline void git_atomic_set(git_atomic *a, int val) #define git_cond_signal(c) (void)0 //pthread_cond_signal(c) #define git_cond_broadcast(c) (void)0 //pthread_cond_broadcast(c) -static inline int git_atomic_inc(git_atomic *a) +GIT_INLINE(int) git_atomic_inc(git_atomic *a) { #ifdef __GNUC__ return __sync_add_and_fetch(&a->val, 1); @@ -47,7 +47,7 @@ static inline int git_atomic_inc(git_atomic *a) #endif } -static inline int git_atomic_dec(git_atomic *a) +GIT_INLINE(int) git_atomic_dec(git_atomic *a) { #ifdef __GNUC__ return __sync_sub_and_fetch(&a->val, 1); @@ -81,12 +81,12 @@ static inline int git_atomic_dec(git_atomic *a) #define git_cond_signal(c) (void)0 #define git_cond_broadcast(c) (void)0 -static inline int git_atomic_inc(git_atomic *a) +GIT_INLINE(int) git_atomic_inc(git_atomic *a) { return ++a->val; } -static inline int git_atomic_dec(git_atomic *a) +GIT_INLINE(int) git_atomic_dec(git_atomic *a) { return --a->val; } diff --git a/src/tree.c b/src/tree.c index 307d41b0d..31b286e69 100644 --- a/src/tree.c +++ b/src/tree.c @@ -41,9 +41,11 @@ int entry_search_cmp(const void *key, const void *array_member) return strcmp(filename, entry->filename); } +#if 0 static int valid_attributes(const int attributes) { return attributes >= 0 && attributes <= MAX_FILEMODE; } +#endif int entry_sort_cmp(const void *a, const void *b) { @@ -56,13 +58,10 @@ int entry_sort_cmp(const void *a, const void *b) entry_b->attr & 040000); } -void git_tree_clear_entries(git_tree *tree) +void git_tree__free(git_tree *tree) { unsigned int i; - if (tree == NULL) - return; - for (i = 0; i < tree->entries.length; ++i) { git_tree_entry *e; e = git_vector_get(&tree->entries, i); @@ -71,32 +70,6 @@ void git_tree_clear_entries(git_tree *tree) free(e); } - git_vector_clear(&tree->entries); - tree->object.modified = 1; -} - - -git_tree *git_tree__new(void) -{ - git_tree *tree; - - tree = git__malloc(sizeof(struct git_tree)); - if (tree == NULL) - return NULL; - - memset(tree, 0x0, sizeof(struct git_tree)); - - if (git_vector_init(&tree->entries, DEFAULT_TREE_SIZE, entry_sort_cmp) < GIT_SUCCESS) { - free(tree); - return NULL; - } - - return tree; -} - -void git_tree__free(git_tree *tree) -{ - git_tree_clear_entries(tree); git_vector_free(&tree->entries); free(tree); } @@ -106,37 +79,6 @@ const git_oid *git_tree_id(git_tree *c) return git_object_id((git_object *)c); } -int git_tree_entry_set_attributes(git_tree_entry *entry, unsigned int attr) -{ - assert(entry && entry->owner); - - if (!valid_attributes(attr)) { - return GIT_ERROR; - } - - entry->attr = attr; - entry->owner->object.modified = 1; - return GIT_SUCCESS; -} - -void git_tree_entry_set_name(git_tree_entry *entry, const char *name) -{ - assert(entry && entry->owner); - - free(entry->filename); - entry->filename = git__strdup(name); - git_vector_sort(&entry->owner->entries); - entry->owner->object.modified = 1; -} - -void git_tree_entry_set_id(git_tree_entry *entry, const git_oid *oid) -{ - assert(entry && entry->owner); - - git_oid_cpy(&entry->oid, oid); - entry->owner->object.modified = 1; -} - unsigned int git_tree_entry_attributes(git_tree_entry *entry) { return entry->attr; @@ -154,15 +96,10 @@ const git_oid *git_tree_entry_id(git_tree_entry *entry) return &entry->oid; } -int git_tree_entry_2object(git_object **object_out, git_tree_entry *entry) +int git_tree_entry_2object(git_object **object_out, git_repository *repo, git_tree_entry *entry) { assert(entry && object_out); - return git_object_lookup(object_out, entry->owner->object.repo, &entry->oid, GIT_OBJ_ANY); -} - -static void sort_entries(git_tree *tree) -{ - git_vector_sort(&tree->entries); + return git_object_lookup(object_out, repo, &entry->oid, GIT_OBJ_ANY); } git_tree_entry *git_tree_entry_byname(git_tree *tree, const char *filename) @@ -171,8 +108,6 @@ git_tree_entry *git_tree_entry_byname(git_tree *tree, const char *filename) assert(tree && filename); - sort_entries(tree); - idx = git_vector_bsearch2(&tree->entries, entry_search_cmp, filename); if (idx == GIT_ENOTFOUND) return NULL; @@ -183,9 +118,6 @@ git_tree_entry *git_tree_entry_byname(git_tree *tree, const char *filename) git_tree_entry *git_tree_entry_byindex(git_tree *tree, int idx) { assert(tree); - - sort_entries(tree); - return git_vector_get(&tree->entries, (unsigned int)idx); } @@ -195,107 +127,12 @@ size_t git_tree_entrycount(git_tree *tree) return tree->entries.length; } -int git_tree_add_entry(git_tree_entry **entry_out, git_tree *tree, const git_oid *id, const char *filename, int attributes) -{ - git_tree_entry *entry; - - assert(tree && id && filename); - if (!valid_attributes(attributes)) { - return GIT_ERROR; - } - - if ((entry = git__malloc(sizeof(git_tree_entry))) == NULL) - return GIT_ENOMEM; - - memset(entry, 0x0, sizeof(git_tree_entry)); - - entry->filename = git__strdup(filename); - git_oid_cpy(&entry->oid, id); - entry->attr = attributes; - entry->owner = tree; - - if (git_vector_insert(&tree->entries, entry) < 0) - return GIT_ENOMEM; - - if (entry_out != NULL) - *entry_out = entry; - - tree->object.modified = 1; - return GIT_SUCCESS; -} - -int git_tree_remove_entry_byindex(git_tree *tree, int idx) -{ - git_tree_entry *remove_ptr; - - assert(tree); - - sort_entries(tree); - - remove_ptr = git_vector_get(&tree->entries, (unsigned int)idx); - if (remove_ptr == NULL) - return GIT_ENOTFOUND; - - free(remove_ptr->filename); - free(remove_ptr); - - tree->object.modified = 1; - - return git_vector_remove(&tree->entries, (unsigned int)idx); -} - -int git_tree_remove_entry_byname(git_tree *tree, const char *filename) -{ - int idx; - - assert(tree && filename); - - sort_entries(tree); - - idx = git_vector_bsearch2(&tree->entries, entry_search_cmp, filename); - if (idx == GIT_ENOTFOUND) - return GIT_ENOTFOUND; - - return git_tree_remove_entry_byindex(tree, idx); -} - -int git_tree__writeback(git_tree *tree, git_odb_source *src) -{ - size_t i; - char filemode[MAX_FILEMODE_BYTES + 1 + 1]; - - assert(tree && src); - - if (tree->entries.length == 0) - return GIT_EMISSINGOBJDATA; - - sort_entries(tree); - - for (i = 0; i < tree->entries.length; ++i) { - git_tree_entry *entry; - - entry = git_vector_get(&tree->entries, i); - - snprintf(filemode, sizeof(filemode), "%o ", entry->attr); - - git__source_write(src, filemode, strlen(filemode)); - git__source_write(src, entry->filename, strlen(entry->filename) + 1); - git__source_write(src, entry->oid.id, GIT_OID_RAWSZ); - } - - return GIT_SUCCESS; -} - - static int tree_parse_buffer(git_tree *tree, char *buffer, char *buffer_end) { - static const size_t avg_entry_size = 40; - unsigned int expected_size; int error = GIT_SUCCESS; - expected_size = (tree->object.source.raw.len / avg_entry_size) + 1; - - git_tree_clear_entries(tree); + if (git_vector_init(&tree->entries, DEFAULT_TREE_SIZE, entry_sort_cmp) < GIT_SUCCESS) + return GIT_ENOMEM; while (buffer < buffer_end) { git_tree_entry *entry; @@ -309,7 +146,6 @@ static int tree_parse_buffer(git_tree *tree, char *buffer, char *buffer_end) if (git_vector_insert(&tree->entries, entry) < GIT_SUCCESS) return GIT_ENOMEM; - entry->owner = tree; entry->attr = strtol(buffer, &buffer, 8); if (*buffer++ != ' ') { @@ -336,16 +172,9 @@ static int tree_parse_buffer(git_tree *tree, char *buffer, char *buffer_end) return error; } -int git_tree__parse(git_tree *tree) +int git_tree__parse(git_tree *tree, git_odb_object *obj) { - char *buffer, *buffer_end; - - assert(tree && tree->object.source.open); - assert(!tree->object.in_memory); - - buffer = tree->object.source.raw.data; - buffer_end = buffer + tree->object.source.raw.len; - - return tree_parse_buffer(tree, buffer, buffer_end); + assert(tree); + return tree_parse_buffer(tree, (char *)obj->raw.data, (char *)obj->raw.data + obj->raw.len); } diff --git a/src/tree.h b/src/tree.h index 78500c471..b4e910a9f 100644 --- a/src/tree.h +++ b/src/tree.h @@ -3,14 +3,13 @@ #include "git2/tree.h" #include "repository.h" +#include "odb.h" #include "vector.h" struct git_tree_entry { unsigned int attr; char *filename; git_oid oid; - - git_tree *owner; }; struct git_tree { @@ -19,8 +18,6 @@ struct git_tree { }; void git_tree__free(git_tree *tree); -git_tree *git_tree__new(void); -int git_tree__parse(git_tree *tree); -int git_tree__writeback(git_tree *tree, git_odb_source *src); +int git_tree__parse(git_tree *tree, git_odb_object *obj); #endif diff --git a/src/util.h b/src/util.h index e0dfd7b20..653b34d02 100644 --- a/src/util.h +++ b/src/util.h @@ -97,6 +97,8 @@ extern char *git__strtok_keep(char *output, char *src, char *delimit); #define STRLEN(str) (sizeof(str) - 1) +#define GIT_OID_LINE_LENGTH(header) (STRLEN(header) + 1 + GIT_OID_HEXSZ + 1) + /* * Realloc the buffer pointed at by variable 'x' so that it can hold * at least 'nr' entries; the number of entries currently allocated diff --git a/tests/t01-rawobj.c b/tests/t01-rawobj.c index 3dfa3c9fe..5db9a79fc 100644 --- a/tests/t01-rawobj.c +++ b/tests/t01-rawobj.c @@ -23,9 +23,16 @@ * Boston, MA 02110-1301, USA. */ #include "test_lib.h" + +#include "odb.h" +#include "hash.h" + #include "t01-data.h" -#include "hash.h" +static int hash_object(git_oid *oid, git_rawobj *obj) +{ + return git_odb_hash(oid, obj->data, obj->len, obj->type); +} BEGIN_TEST(oid0, "validate size of oid objects") git_oid out; @@ -497,28 +504,28 @@ BEGIN_TEST(objhash0, "hash junk data") /* invalid types: */ junk_obj.data = some_data; - must_fail(git_rawobj_hash(&id, &junk_obj)); + must_fail(hash_object(&id, &junk_obj)); junk_obj.type = GIT_OBJ__EXT1; - must_fail(git_rawobj_hash(&id, &junk_obj)); + must_fail(hash_object(&id, &junk_obj)); junk_obj.type = GIT_OBJ__EXT2; - must_fail(git_rawobj_hash(&id, &junk_obj)); + must_fail(hash_object(&id, &junk_obj)); junk_obj.type = GIT_OBJ_OFS_DELTA; - must_fail(git_rawobj_hash(&id, &junk_obj)); + must_fail(hash_object(&id, &junk_obj)); junk_obj.type = GIT_OBJ_REF_DELTA; - must_fail(git_rawobj_hash(&id, &junk_obj)); + must_fail(hash_object(&id, &junk_obj)); /* data can be NULL only if len is zero: */ junk_obj.type = GIT_OBJ_BLOB; junk_obj.data = NULL; - must_pass(git_rawobj_hash(&id, &junk_obj)); + must_pass(hash_object(&id, &junk_obj)); must_be_true(git_oid_cmp(&id, &id_zero) == 0); junk_obj.len = 1; - must_fail(git_rawobj_hash(&id, &junk_obj)); + must_fail(hash_object(&id, &junk_obj)); END_TEST BEGIN_TEST(objhash1, "hash a commit object") @@ -526,7 +533,7 @@ BEGIN_TEST(objhash1, "hash a commit object") must_pass(git_oid_mkstr(&id1, commit_id)); - must_pass(git_rawobj_hash(&id2, &commit_obj)); + must_pass(hash_object(&id2, &commit_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); END_TEST @@ -536,7 +543,7 @@ BEGIN_TEST(objhash2, "hash a tree object") must_pass(git_oid_mkstr(&id1, tree_id)); - must_pass(git_rawobj_hash(&id2, &tree_obj)); + must_pass(hash_object(&id2, &tree_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); END_TEST @@ -546,7 +553,7 @@ BEGIN_TEST(objhash3, "hash a tag object") must_pass(git_oid_mkstr(&id1, tag_id)); - must_pass(git_rawobj_hash(&id2, &tag_obj)); + must_pass(hash_object(&id2, &tag_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); END_TEST @@ -556,7 +563,7 @@ BEGIN_TEST(objhash4, "hash a zero-length object") must_pass(git_oid_mkstr(&id1, zero_id)); - must_pass(git_rawobj_hash(&id2, &zero_obj)); + must_pass(hash_object(&id2, &zero_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); END_TEST @@ -566,7 +573,7 @@ BEGIN_TEST(objhash5, "hash an one-byte long object") must_pass(git_oid_mkstr(&id1, one_id)); - must_pass(git_rawobj_hash(&id2, &one_obj)); + must_pass(hash_object(&id2, &one_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); END_TEST @@ -576,7 +583,7 @@ BEGIN_TEST(objhash6, "hash a two-byte long object") must_pass(git_oid_mkstr(&id1, two_id)); - must_pass(git_rawobj_hash(&id2, &two_obj)); + must_pass(hash_object(&id2, &two_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); END_TEST @@ -586,7 +593,7 @@ BEGIN_TEST(objhash7, "hash an object several bytes long") must_pass(git_oid_mkstr(&id1, some_id)); - must_pass(git_rawobj_hash(&id2, &some_obj)); + must_pass(hash_object(&id2, &some_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); END_TEST diff --git a/tests/t02-objread.c b/tests/t02-objread.c index 2a9d130c4..85b03b026 100644 --- a/tests/t02-objread.c +++ b/tests/t02-objread.c @@ -24,6 +24,7 @@ */ #include "test_lib.h" #include "test_helpers.h" +#include "odb.h" #include "t02-data.h" #include "t02-oids.h" @@ -50,16 +51,16 @@ END_TEST BEGIN_TEST(readloose0, "read a loose commit") git_odb *db; git_oid id; - git_rawobj obj; + git_odb_object *obj; must_pass(write_object_files(odb_dir, &commit)); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id, commit.id)); must_pass(git_odb_read(&obj, db, &id)); - must_pass(cmp_objects(&obj, &commit)); + must_pass(cmp_objects((git_rawobj *)&obj->raw, &commit)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(odb_dir, &commit)); END_TEST @@ -67,16 +68,16 @@ END_TEST BEGIN_TEST(readloose1, "read a loose tree") git_odb *db; git_oid id; - git_rawobj obj; + git_odb_object *obj; must_pass(write_object_files(odb_dir, &tree)); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id, tree.id)); must_pass(git_odb_read(&obj, db, &id)); - must_pass(cmp_objects(&obj, &tree)); + must_pass(cmp_objects((git_rawobj *)&obj->raw, &tree)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(odb_dir, &tree)); END_TEST @@ -84,16 +85,16 @@ END_TEST BEGIN_TEST(readloose2, "read a loose tag") git_odb *db; git_oid id; - git_rawobj obj; + git_odb_object *obj; must_pass(write_object_files(odb_dir, &tag)); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id, tag.id)); must_pass(git_odb_read(&obj, db, &id)); - must_pass(cmp_objects(&obj, &tag)); + must_pass(cmp_objects((git_rawobj *)&obj->raw, &tag)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(odb_dir, &tag)); END_TEST @@ -101,16 +102,16 @@ END_TEST BEGIN_TEST(readloose3, "read a loose zero-bytes object") git_odb *db; git_oid id; - git_rawobj obj; + git_odb_object *obj; must_pass(write_object_files(odb_dir, &zero)); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id, zero.id)); must_pass(git_odb_read(&obj, db, &id)); - must_pass(cmp_objects(&obj, &zero)); + must_pass(cmp_objects((git_rawobj *)&obj->raw, &zero)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(odb_dir, &zero)); END_TEST @@ -118,16 +119,16 @@ END_TEST BEGIN_TEST(readloose4, "read a one-byte long loose object") git_odb *db; git_oid id; - git_rawobj obj; + git_odb_object *obj; must_pass(write_object_files(odb_dir, &one)); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id, one.id)); must_pass(git_odb_read(&obj, db, &id)); - must_pass(cmp_objects(&obj, &one)); + must_pass(cmp_objects(&obj->raw, &one)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(odb_dir, &one)); END_TEST @@ -135,16 +136,16 @@ END_TEST BEGIN_TEST(readloose5, "read a two-bytes long loose object") git_odb *db; git_oid id; - git_rawobj obj; + git_odb_object *obj; must_pass(write_object_files(odb_dir, &two)); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id, two.id)); must_pass(git_odb_read(&obj, db, &id)); - must_pass(cmp_objects(&obj, &two)); + must_pass(cmp_objects(&obj->raw, &two)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(odb_dir, &two)); END_TEST @@ -152,16 +153,16 @@ END_TEST BEGIN_TEST(readloose6, "read a loose object which is several bytes long") git_odb *db; git_oid id; - git_rawobj obj; + git_odb_object *obj; must_pass(write_object_files(odb_dir, &some)); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id, some.id)); must_pass(git_odb_read(&obj, db, &id)); - must_pass(cmp_objects(&obj, &some)); + must_pass(cmp_objects(&obj->raw, &some)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(odb_dir, &some)); END_TEST @@ -174,13 +175,13 @@ BEGIN_TEST(readpack0, "read several packed objects") for (i = 0; i < ARRAY_SIZE(packed_objects); ++i) { git_oid id; - git_rawobj obj; + git_odb_object *obj; must_pass(git_oid_mkstr(&id, packed_objects[i])); must_be_true(git_odb_exists(db, &id) == 1); must_pass(git_odb_read(&obj, db, &id)); - git_rawobj_close(&obj); + git_odb_object_close(obj); } git_odb_close(db); @@ -194,17 +195,19 @@ BEGIN_TEST(readheader0, "read only the header of several packed objects") for (i = 0; i < ARRAY_SIZE(packed_objects); ++i) { git_oid id; - git_rawobj obj, header; + git_odb_object *obj; + size_t len; + git_otype type; must_pass(git_oid_mkstr(&id, packed_objects[i])); must_pass(git_odb_read(&obj, db, &id)); - must_pass(git_odb_read_header(&header, db, &id)); + must_pass(git_odb_read_header(&len, &type, db, &id)); - must_be_true(obj.len == header.len); - must_be_true(obj.type == header.type); + must_be_true(obj->raw.len == len); + must_be_true(obj->raw.type == type); - git_rawobj_close(&obj); + git_odb_object_close(obj); } git_odb_close(db); @@ -218,19 +221,21 @@ BEGIN_TEST(readheader1, "read only the header of several loose objects") for (i = 0; i < ARRAY_SIZE(loose_objects); ++i) { git_oid id; - git_rawobj obj, header; + git_odb_object *obj; + size_t len; + git_otype type; must_pass(git_oid_mkstr(&id, loose_objects[i])); must_be_true(git_odb_exists(db, &id) == 1); must_pass(git_odb_read(&obj, db, &id)); - must_pass(git_odb_read_header(&header, db, &id)); + must_pass(git_odb_read_header(&len, &type, db, &id)); - must_be_true(obj.len == header.len); - must_be_true(obj.type == header.type); + must_be_true(obj->raw.len == len); + must_be_true(obj->raw.type == type); - git_rawobj_close(&obj); + git_odb_object_close(obj); } git_odb_close(db); diff --git a/tests/t03-objwrite.c b/tests/t03-objwrite.c index 10c6c7f1a..773887397 100644 --- a/tests/t03-objwrite.c +++ b/tests/t03-objwrite.c @@ -24,6 +24,7 @@ */ #include "test_lib.h" #include "fileops.h" +#include "odb.h" static char *odb_dir = "test-objects"; #include "t03-data.h" @@ -80,23 +81,39 @@ static int remove_object_files(object_data *d) return 0; } +static int streaming_write(git_oid *oid, git_odb *odb, git_rawobj *raw) +{ + git_odb_stream *stream; + int error; + + if ((error = git_odb_open_wstream(&stream, odb, raw->len, raw->type)) < GIT_SUCCESS) + return error; + + stream->write(stream, raw->data, raw->len); + + error = stream->finalize_write(oid, stream); + stream->free(stream); + + return error; +} + BEGIN_TEST(write0, "write loose commit object") git_odb *db; git_oid id1, id2; - git_rawobj obj; + git_odb_object *obj; must_pass(make_odb_dir()); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id1, commit.id)); - must_pass(git_odb_write(&id2, db, &commit_obj)); + must_pass(streaming_write(&id2, db, &commit_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); must_pass(check_object_files(&commit)); must_pass(git_odb_read(&obj, db, &id1)); - must_pass(cmp_objects(&obj, &commit_obj)); + must_pass(cmp_objects(&obj->raw, &commit_obj)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(&commit)); END_TEST @@ -104,20 +121,20 @@ END_TEST BEGIN_TEST(write1, "write loose tree object") git_odb *db; git_oid id1, id2; - git_rawobj obj; + git_odb_object *obj; must_pass(make_odb_dir()); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id1, tree.id)); - must_pass(git_odb_write(&id2, db, &tree_obj)); + must_pass(streaming_write(&id2, db, &tree_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); must_pass(check_object_files(&tree)); must_pass(git_odb_read(&obj, db, &id1)); - must_pass(cmp_objects(&obj, &tree_obj)); + must_pass(cmp_objects(&obj->raw, &tree_obj)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(&tree)); END_TEST @@ -125,20 +142,20 @@ END_TEST BEGIN_TEST(write2, "write loose tag object") git_odb *db; git_oid id1, id2; - git_rawobj obj; + git_odb_object *obj; must_pass(make_odb_dir()); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id1, tag.id)); - must_pass(git_odb_write(&id2, db, &tag_obj)); + must_pass(streaming_write(&id2, db, &tag_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); must_pass(check_object_files(&tag)); must_pass(git_odb_read(&obj, db, &id1)); - must_pass(cmp_objects(&obj, &tag_obj)); + must_pass(cmp_objects(&obj->raw, &tag_obj)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(&tag)); END_TEST @@ -146,20 +163,20 @@ END_TEST BEGIN_TEST(write3, "write zero-length object") git_odb *db; git_oid id1, id2; - git_rawobj obj; + git_odb_object *obj; must_pass(make_odb_dir()); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id1, zero.id)); - must_pass(git_odb_write(&id2, db, &zero_obj)); + must_pass(streaming_write(&id2, db, &zero_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); must_pass(check_object_files(&zero)); must_pass(git_odb_read(&obj, db, &id1)); - must_pass(cmp_objects(&obj, &zero_obj)); + must_pass(cmp_objects(&obj->raw, &zero_obj)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(&zero)); END_TEST @@ -167,20 +184,20 @@ END_TEST BEGIN_TEST(write4, "write one-byte long object") git_odb *db; git_oid id1, id2; - git_rawobj obj; + git_odb_object *obj; must_pass(make_odb_dir()); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id1, one.id)); - must_pass(git_odb_write(&id2, db, &one_obj)); + must_pass(streaming_write(&id2, db, &one_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); must_pass(check_object_files(&one)); must_pass(git_odb_read(&obj, db, &id1)); - must_pass(cmp_objects(&obj, &one_obj)); + must_pass(cmp_objects(&obj->raw, &one_obj)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(&one)); END_TEST @@ -188,20 +205,20 @@ END_TEST BEGIN_TEST(write5, "write two-byte long object") git_odb *db; git_oid id1, id2; - git_rawobj obj; + git_odb_object *obj; must_pass(make_odb_dir()); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id1, two.id)); - must_pass(git_odb_write(&id2, db, &two_obj)); + must_pass(streaming_write(&id2, db, &two_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); must_pass(check_object_files(&two)); must_pass(git_odb_read(&obj, db, &id1)); - must_pass(cmp_objects(&obj, &two_obj)); + must_pass(cmp_objects(&obj->raw, &two_obj)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(&two)); END_TEST @@ -209,20 +226,20 @@ END_TEST BEGIN_TEST(write6, "write an object which is several bytes long") git_odb *db; git_oid id1, id2; - git_rawobj obj; + git_odb_object *obj; must_pass(make_odb_dir()); must_pass(git_odb_open(&db, odb_dir)); must_pass(git_oid_mkstr(&id1, some.id)); - must_pass(git_odb_write(&id2, db, &some_obj)); + must_pass(streaming_write(&id2, db, &some_obj)); must_be_true(git_oid_cmp(&id1, &id2) == 0); must_pass(check_object_files(&some)); must_pass(git_odb_read(&obj, db, &id1)); - must_pass(cmp_objects(&obj, &some_obj)); + must_pass(cmp_objects(&obj->raw, &some_obj)); - git_rawobj_close(&obj); + git_odb_object_close(obj); git_odb_close(db); must_pass(remove_object_files(&some)); END_TEST diff --git a/tests/t04-commit.c b/tests/t04-commit.c index 855cf9859..1140d3319 100644 --- a/tests/t04-commit.c +++ b/tests/t04-commit.c @@ -407,39 +407,42 @@ This is a commit created in memory and it will be written back to disk\n" static const char *tree_oid = "1810dff58d8a660512d4832e740f692884338ccd"; + BEGIN_TEST(write0, "write a new commit object from memory to disk") git_repository *repo; - git_commit *commit, *parent; - git_tree *tree; - git_oid id; + git_commit *commit; + git_oid tree_id, parent_id, commit_id; const git_signature *author, *committer; /* char hex_oid[41]; */ must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); - /* Create commit in memory */ - must_pass(git_commit_new(&commit, repo)); - /* Add new parent */ - git_oid_mkstr(&id, commit_ids[4]); - must_pass(git_commit_lookup(&parent, repo, &id)); + git_oid_mkstr(&tree_id, tree_oid); + git_oid_mkstr(&parent_id, commit_ids[4]); - git_commit_add_parent(commit, parent); - - /* Set other attributes */ + /* create signatures */ committer = git_signature_new(COMMITTER_NAME, COMMITTER_EMAIL, 123456789, 60); must_be_true(committer != NULL); author = git_signature_new(COMMITTER_NAME, COMMITTER_EMAIL, 987654321, 90); must_be_true(author != NULL); - git_commit_set_committer(commit, committer); - git_commit_set_author(commit, author); - git_commit_set_message(commit, COMMIT_MESSAGE); + must_pass(git_commit_create_v( + &commit_id, /* out id */ + repo, + NULL, /* do not update the HEAD */ + author, + committer, + COMMIT_MESSAGE, + &tree_id, + 1, &parent_id)); git_signature_free((git_signature *)committer); git_signature_free((git_signature *)author); + must_pass(git_commit_lookup(&commit, repo, &commit_id)); + /* Check attributes were set correctly */ author = git_commit_author(commit); must_be_true(author != NULL); @@ -457,47 +460,6 @@ BEGIN_TEST(write0, "write a new commit object from memory to disk") must_be_true(strcmp(git_commit_message(commit), COMMIT_MESSAGE) == 0); - /* add new tree */ - git_oid_mkstr(&id, tree_oid); - must_pass(git_tree_lookup(&tree, repo, &id)); - - git_commit_set_tree(commit, tree); - - /* Test it has no OID */ - must_be_true(git_commit_id(commit) == NULL); - - /* Write to disk */ - must_pass(git_object_write((git_object *)commit)); - - must_pass(remove_loose_object(REPOSITORY_FOLDER, (git_object *)commit)); - - git_repository_free(repo); -END_TEST - -BEGIN_TEST(write1, "load a commit object, modify it and write it back") - git_repository *repo; - git_oid id; - git_commit *commit, *parent; - const char *message; - /* char hex_oid[41]; */ - - must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); - - git_oid_mkstr(&id, commit_ids[0]); - - must_pass(git_commit_lookup(&commit, repo, &id)); - - message = git_commit_message(commit); - - git_commit_set_message(commit, "This is a new test message. Cool!\n"); - - git_oid_mkstr(&id, commit_ids[4]); - must_pass(git_commit_lookup(&parent, repo, &id)); - - git_commit_add_parent(commit, parent); - - must_pass(git_object_write((git_object *)commit)); - must_pass(remove_loose_object(REPOSITORY_FOLDER, (git_object *)commit)); git_repository_free(repo); @@ -509,6 +471,7 @@ BEGIN_SUITE(commit) ADD_TEST(parse1); ADD_TEST(parse2); ADD_TEST(details0); + ADD_TEST(write0); - ADD_TEST(write1); + //ADD_TEST(write1); END_SUITE diff --git a/tests/t08-tag.c b/tests/t08-tag.c index c6789266c..70eeb28a6 100644 --- a/tests/t08-tag.c +++ b/tests/t08-tag.c @@ -61,27 +61,62 @@ BEGIN_TEST(read0, "read and parse a tag from the repository") git_repository_free(repo); END_TEST -BEGIN_TEST(write0, "write back a tag to the repository") - git_oid id; + +#define TAGGER_NAME "Vicent Marti" +#define TAGGER_EMAIL "vicent@github.com" +#define TAGGER_MESSAGE "This is my tag.\n\nThere are many tags, but this one is mine\n" + +BEGIN_TEST(write0, "write a tag to the repository and read it again") git_repository *repo; git_tag *tag; + git_oid target_id, tag_id; + const git_signature *tagger; + git_reference *ref_tag; + /* char hex_oid[41]; */ must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); - git_oid_mkstr(&id, tag1_id); + git_oid_mkstr(&target_id, tagged_commit); - must_pass(git_tag_lookup(&tag, repo, &id)); + /* create signatures */ + tagger = git_signature_new(TAGGER_NAME, TAGGER_EMAIL, 123456789, 60); + must_be_true(tagger != NULL); - git_tag_set_name(tag, "This is a different tag LOL"); + must_pass(git_tag_create( + &tag_id, /* out id */ + repo, + "the-tag", /* do not update the HEAD */ + &target_id, + GIT_OBJ_COMMIT, + tagger, + TAGGER_MESSAGE)); + + git_signature_free((git_signature *)tagger); + + must_pass(git_tag_lookup(&tag, repo, &tag_id)); + + /* Check attributes were set correctly */ + tagger = git_tag_tagger(tag); + must_be_true(tagger != NULL); + must_be_true(strcmp(tagger->name, TAGGER_NAME) == 0); + must_be_true(strcmp(tagger->email, TAGGER_EMAIL) == 0); + must_be_true(tagger->when.time == 123456789); + must_be_true(tagger->when.offset == 60); + + must_be_true(strcmp(git_tag_message(tag), TAGGER_MESSAGE) == 0); + + must_pass(git_reference_lookup(&ref_tag, repo, "refs/tags/the-tag")); + must_be_true(git_oid_cmp(git_reference_oid(ref_tag), &tag_id) == 0); + must_pass(git_reference_delete(ref_tag)); - must_pass(git_object_write((git_object *)tag)); must_pass(remove_loose_object(REPOSITORY_FOLDER, (git_object *)tag)); git_repository_free(repo); + END_TEST BEGIN_SUITE(tag) ADD_TEST(read0); - ADD_TEST(write0); + ADD_TEST(write0); END_SUITE diff --git a/tests/t09-tree.c b/tests/t09-tree.c index dfd266d16..6c1b2e643 100644 --- a/tests/t09-tree.c +++ b/tests/t09-tree.c @@ -75,93 +75,15 @@ BEGIN_TEST(read1, "read a tree from the repository") must_be_true(strcmp(git_tree_entry_name(entry), "README") == 0); - must_pass(git_tree_entry_2object(&obj, entry)); + must_pass(git_tree_entry_2object(&obj, repo, entry)); git_repository_free(repo); END_TEST -BEGIN_TEST(write0, "add a new entry to a tree and write it back to disk") - const unsigned int entry_count = 128; - - git_repository *repo; - git_tree *tree; - unsigned int i; - git_oid entry_id; - - must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); - must_pass(git_tree_new(&tree, repo)); - - git_oid_mkstr(&entry_id, tree_oid); - for (i = 0; i < entry_count; ++i) { - char filename[32]; - git_tree_entry *ent = NULL; - - sprintf(filename, "file%d.txt", i); - must_pass(git_tree_add_entry(&ent, tree, &entry_id, filename, 040000)); - must_be_true(ent != NULL); - } - - must_be_true(git_tree_entrycount(tree) == entry_count); - must_pass(git_object_write((git_object *)tree)); - must_pass(remove_loose_object(REPOSITORY_FOLDER, (git_object *)tree)); - - git_repository_free(repo); -END_TEST - -BEGIN_TEST(write1, "add several entries in-memory and validate that they exist; write back to disk") - git_oid id; - git_repository *repo; - git_tree *tree; - git_tree_entry *entry; - unsigned int i; - /* char hex_oid[41]; */ - - must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); - - git_oid_mkstr(&id, tree_oid); - - must_pass(git_tree_lookup(&tree, repo, &id)); - - must_be_true(git_tree_entrycount(tree) == 3); - - /* check there is NP if we don't want the - * created entry back */ - git_tree_add_entry(NULL, tree, &id, "zzz_test_entry.dat", 0); - git_tree_add_entry(NULL, tree, &id, "01_test_entry.txt", 0); - - must_be_true(git_tree_entrycount(tree) == 5); - - entry = git_tree_entry_byindex(tree, 0); - must_be_true(strcmp(git_tree_entry_name(entry), "01_test_entry.txt") == 0); - - entry = git_tree_entry_byindex(tree, 4); - must_be_true(strcmp(git_tree_entry_name(entry), "zzz_test_entry.dat") == 0); - - must_pass(git_tree_remove_entry_byname(tree, "README")); - must_be_true(git_tree_entrycount(tree) == 4); - - for (i = 0; i < git_tree_entrycount(tree); ++i) { - entry = git_tree_entry_byindex(tree, i); - must_be_true(strcmp(git_tree_entry_name(entry), "README") != 0); - } - - must_pass(git_object_write((git_object *)tree)); - -/* - git_oid_fmt(hex_oid, git_tree_id(tree)); - hex_oid[40] = 0; - printf("TREE New SHA1: %s\n", hex_oid); -*/ - - must_pass(remove_loose_object(REPOSITORY_FOLDER, (git_object *)tree)); - git_repository_free(repo); -END_TEST - - BEGIN_SUITE(tree) ADD_TEST(read0); ADD_TEST(read1); - ADD_TEST(write0); - ADD_TEST(write1); +// ADD_TEST(write0); /* TODO THREADSAFE */ +// ADD_TEST(write1); END_SUITE diff --git a/tests/t11-sqlite.c b/tests/t11-sqlite.c index ba9065e76..fecf7886f 100644 --- a/tests/t11-sqlite.c +++ b/tests/t11-sqlite.c @@ -31,7 +31,7 @@ #include "git2/odb_backend.h" -static int cmp_objects(git_rawobj *o1, git_rawobj *o2) +static int cmp_objects(raw_object *o1, raw_object *o2) { if (o1->type != o2->type) return -1; @@ -62,7 +62,7 @@ static git_odb *open_sqlite_odb(void) #define TEST_WRITE(PTR) {\ git_odb *db; \ git_oid id1, id2; \ - git_rawobj obj; \ + raw_object obj; \ db = open_sqlite_odb(); \ must_be_true(db != NULL); \ must_pass(git_oid_mkstr(&id1, PTR.id)); \ diff --git a/tests/test_helpers.h b/tests/test_helpers.h index 97b81ab40..9a24ebccf 100644 --- a/tests/test_helpers.h +++ b/tests/test_helpers.h @@ -29,6 +29,8 @@ #include "test_lib.h" #include +#include "odb.h" + #define TEST_REPOSITORY_NAME "testrepo.git" #define REPOSITORY_FOLDER TEST_RESOURCES "/" TEST_REPOSITORY_NAME "/" #define ODB_FOLDER (REPOSITORY_FOLDER "objects/") From 99baacfb5687c266df78be4442d30f2f2dc19a28 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Mon, 21 Mar 2011 19:27:45 +0200 Subject: [PATCH 31/81] Fix MSVC warnings --- src/cache.c | 2 +- src/commit.c | 6 +++--- tests/t13-threads.c | 7 ------- 3 files changed, 4 insertions(+), 11 deletions(-) diff --git a/src/cache.c b/src/cache.c index fdf5c1b80..fd42e2c5b 100644 --- a/src/cache.c +++ b/src/cache.c @@ -78,7 +78,7 @@ void *git_cache_get(git_cache *cache, const git_oid *oid) { const uint32_t *hash; size_t i, pos, found = 0; - cache_node *node; + cache_node *node = NULL; hash = (const uint32_t *)oid->id; diff --git a/src/commit.c b/src/commit.c index 2670c2e0a..d5d6ebd8a 100644 --- a/src/commit.c +++ b/src/commit.c @@ -101,7 +101,7 @@ int git_commit_create_v( oid, repo, update_ref, author, committer, message, tree_oid, parent_count, oids); - free(oids); + free((void *)oids); return error; } @@ -132,7 +132,7 @@ int git_commit_create_ov( git_object_id((git_object *)tree), parent_count, oids); - free(oids); + free((void *)oids); return error; } @@ -160,7 +160,7 @@ int git_commit_create_o( git_object_id((git_object *)tree), parent_count, oids); - free(oids); + free((void *)oids); return error; } diff --git a/tests/t13-threads.c b/tests/t13-threads.c index 62ca73ac2..3888b70ce 100644 --- a/tests/t13-threads.c +++ b/tests/t13-threads.c @@ -32,13 +32,6 @@ typedef struct { unsigned int __dummy; } ttest_obj; -void *cache0_thread(void *data) -{ - git_cache *cache = (git_cache*)data; - unsigned int num; - -} - BEGIN_TEST(cache0, "run several threads polling the cache at the same time") END_TEST From 70236bab2d64a4d34203b4e2bbc5d8f518ba618d Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Mon, 21 Mar 2011 19:39:06 +0200 Subject: [PATCH 32/81] Update CMakeLists.txt to support threadsafe --- CMakeLists.txt | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 1bd243099..acac2a6de 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -62,12 +62,21 @@ SET(INSTALL_INC include CACHE PATH "Where to install headers to.") # Build options OPTION (BUILD_SHARED_LIBS "Build Shared Library (OFF for Static)" ON) OPTION (BUILD_TESTS "Build Tests" ON) +OPTION (THREADSAFE "Build libgit2 as threadsafe" OFF) # Build Release by default IF (NOT CMAKE_BUILD_TYPE) SET(CMAKE_BUILD_TYPE "Release" CACHE STRING "Choose the type of build, options are: Debug Release RelWithDebInfo MinSizeRel." FORCE) ENDIF () +IF (THREADSAFE) + IF (NOT WIN32) + find_package(Threads REQUIRED) + ENDIF() + + ADD_DEFINITIONS(-DGIT_THREADS) +ENDIF() + # Collect sourcefiles FILE(GLOB SRC src/*.c src/backends/*.c) FILE(GLOB SRC_ZLIB deps/zlib/*.c) @@ -79,9 +88,6 @@ FILE(GLOB SRC_H include/git2/*.h) IF (WIN32 AND NOT CYGWIN) ADD_DEFINITIONS(-DWIN32 -D_DEBUG -D_LIB) FILE(GLOB SRC_PLAT src/win32/*.c) - IF (MINGW) - SET(PTHREAD_LIBRARY pthread) - ENDIF () ENDIF () # Specify sha1 implementation @@ -97,7 +103,7 @@ ENDIF () # Compile and link libgit2 ADD_LIBRARY(git2 ${SRC} ${SRC_PLAT} ${SRC_SHA1} ${SRC_ZLIB}) -TARGET_LINK_LIBRARIES(git2 ${LIB_SHA1} ${PTHREAD_LIBRARY} ${SQLITE3_LIBRARIES}) +TARGET_LINK_LIBRARIES(git2 ${LIB_SHA1} ${CMAKE_THREAD_LIBS_INIT} ${SQLITE3_LIBRARIES}) SET_TARGET_PROPERTIES(git2 PROPERTIES VERSION ${LIBGIT2_VERSION_STRING}) SET_TARGET_PROPERTIES(git2 PROPERTIES SOVERSION ${LIBGIT2_VERSION_MAJOR}) @@ -121,7 +127,7 @@ IF (BUILD_TESTS) FILE(GLOB SRC_TEST tests/t??-*.c) ADD_EXECUTABLE(libgit2_test tests/test_main.c tests/test_lib.c tests/test_helpers.c ${SRC} ${SRC_PLAT} ${SRC_SHA1} ${SRC_TEST} ${SRC_ZLIB}) - TARGET_LINK_LIBRARIES(libgit2_test ${LIB_SHA1} ${PTHREAD_LIBRARY} ${SQLITE3_LIBRARIES}) + TARGET_LINK_LIBRARIES(libgit2_test ${LIB_SHA1} ${CMAKE_THREAD_LIBS_INIT} ${SQLITE3_LIBRARIES}) ADD_TEST(libgit2_test libgit2_test) ENDIF () From 567fc1d20c734c34dd1c4c19be12abcc1622aa5d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Tue, 22 Mar 2011 12:47:53 +0100 Subject: [PATCH 33/81] refs loose_lookup: also free the buffer on success Free the ref_file buffer at the end of the function also on success. This fixes a small memory leak. --- src/refs.c | 1 + 1 file changed, 1 insertion(+) diff --git a/src/refs.c b/src/refs.c index 40b80ec9b..8e2965528 100644 --- a/src/refs.c +++ b/src/refs.c @@ -333,6 +333,7 @@ static int loose_lookup( ref->mtime = ref_time; *ref_out = ref; + gitfo_free_buf(&ref_file); return GIT_SUCCESS; cleanup: From b932ef5b3d77b26b8f558bafa3ee416595279dce Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Mon, 21 Mar 2011 19:44:43 +0200 Subject: [PATCH 34/81] Fix MSVC warnings when building threads --- src/thread-utils.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/thread-utils.h b/src/thread-utils.h index e542639c8..20d6022ea 100644 --- a/src/thread-utils.h +++ b/src/thread-utils.h @@ -5,7 +5,11 @@ /* Common operations even if threading has been disabled */ typedef struct { +#if defined(_MSC_VER) + volatile long val; +#else volatile int val; +#endif } git_atomic; GIT_INLINE(void) git_atomic_set(git_atomic *a, int val) From 1881f0783b6dd23fb40ba0b43a4ec98bb08a637c Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Mon, 21 Mar 2011 20:28:02 +0200 Subject: [PATCH 35/81] Add getters for `git_odb_object` --- include/git2/odb.h | 42 ++++++++++++++++++++++++++++++++++++++++++ src/odb.c | 20 ++++++++++++++++++++ 2 files changed, 62 insertions(+) diff --git a/include/git2/odb.h b/include/git2/odb.h index 8926b446c..1a0f0e64c 100644 --- a/include/git2/odb.h +++ b/include/git2/odb.h @@ -235,6 +235,48 @@ GIT_EXTERN(int) git_odb_hash(git_oid *id, const void *data, size_t len, git_otyp */ GIT_EXTERN(void) git_odb_object_close(git_odb_object *object); +/** + * Return the OID of an ODB object + * + * This is the OID from which the object was read from + * + * @param object the object + * @return a pointer to the OID + */ +GIT_EXTERN(const git_oid *) git_odb_object_id(git_odb_object *object); + +/** + * Return the data of an ODB object + * + * This is the uncompressed, raw data as read from the ODB, + * without the leading header. + * + * This pointer is owned by the object and shall not be free'd. + * + * @param object the object + * @return a pointer to the data + */ +GIT_EXTERN(const void *) git_odb_object_data(git_odb_object *object); + +/** + * Return the size of an ODB object + * + * This is the real size of the `data` buffer, not the + * actual size of the object. + * + * @param object the object + * @return the size + */ +GIT_EXTERN(size_t) git_odb_object_size(git_odb_object *object); + +/** + * Return the type of an ODB object + * + * @param object the object + * @return the type + */ +GIT_EXTERN(git_otype) git_odb_object_type(git_odb_object *object); + /** @} */ GIT_END_DECL #endif diff --git a/src/odb.c b/src/odb.c index 9aeaa8a23..d825fd95c 100644 --- a/src/odb.c +++ b/src/odb.c @@ -109,6 +109,26 @@ static void free_odb_object(void *o) } } +const git_oid *git_odb_object_id(git_odb_object *object) +{ + return &object->cached.oid; +} + +const void *git_odb_object_data(git_odb_object *object) +{ + return object->raw.data; +} + +size_t git_odb_object_size(git_odb_object *object) +{ + return object->raw.len; +} + +git_otype git_odb_object_type(git_odb_object *object) +{ + return object->raw.type; +} + void git_odb_object_close(git_odb_object *object) { git_cached_obj_decref((git_cached_obj *)object, &free_odb_object); From 21d73e7195a427b39c1c3742c85254e8384f5802 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 22 Mar 2011 20:26:01 +0200 Subject: [PATCH 36/81] Always free the parents of a revwalk commit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Thanks to Carlos Martín Nieto for spotting this. --- src/revwalk.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/src/revwalk.c b/src/revwalk.c index a9d4f8734..73bb060f5 100644 --- a/src/revwalk.c +++ b/src/revwalk.c @@ -482,11 +482,22 @@ int git_revwalk_new(git_revwalk **revwalk_out, git_repository *repo) void git_revwalk_free(git_revwalk *walk) { unsigned int i; + const void *_unused; + commit_object *commit; if (walk == NULL) return; git_revwalk_reset(walk); + + /* if the parent has more than PARENTS_PER_COMMIT parents, + * we had to allocate a separate array for those parents. + * make sure it's being free'd */ + GIT_HASHTABLE_FOREACH(walk->commits, _unused, commit, { + if (commit->out_degree > PARENTS_PER_COMMIT) + free(commit->parents); + }); + git_hashtable_free(walk->commits); git_pqueue_free(&walk->iterator_time); From fe1920206b7fb780486dcef14d0d6f01835b430d Mon Sep 17 00:00:00 2001 From: Tim Branyen Date: Mon, 21 Mar 2011 23:27:54 -0400 Subject: [PATCH 37/81] Corrected the header include directory in doxygen conf --- api.doxygen | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/api.doxygen b/api.doxygen index d37814a1b..b812add85 100644 --- a/api.doxygen +++ b/api.doxygen @@ -1,6 +1,6 @@ PROJECT_NAME = libgit2 -INPUT = src/git2 +INPUT = include/git2 QUIET = YES RECURSIVE = YES FILE_PATTERNS = *.h From 56d8ca266c92e51a1b38ac68b7aa6a24193f5812 Mon Sep 17 00:00:00 2001 From: nulltoken Date: Sun, 20 Mar 2011 18:36:25 +0100 Subject: [PATCH 38/81] Switch from time_t to git_time_t git_time_t is defined as a signed 64 integer. This allows a true predictable multiplatform behavior. --- include/git2/commit.h | 2 +- include/git2/signature.h | 2 +- include/git2/types.h | 10 +++++----- src/commit.c | 2 +- src/index.c | 12 ++++++------ src/odb_pack.c | 4 ++-- src/signature.c | 2 +- tests/t04-commit.c | 2 +- tests/t06-index.c | 2 +- 9 files changed, 19 insertions(+), 19 deletions(-) diff --git a/include/git2/commit.h b/include/git2/commit.h index ba18a5b39..8306eb152 100644 --- a/include/git2/commit.h +++ b/include/git2/commit.h @@ -83,7 +83,7 @@ GIT_EXTERN(const char *) git_commit_message(git_commit *commit); * @param commit a previously loaded commit. * @return the time of a commit */ -GIT_EXTERN(time_t) git_commit_time(git_commit *commit); +GIT_EXTERN(git_time_t) git_commit_time(git_commit *commit); /** * Get the commit timezone offset (i.e. committer's preferred timezone) of a commit. diff --git a/include/git2/signature.h b/include/git2/signature.h index 96275aa07..200397ab3 100644 --- a/include/git2/signature.h +++ b/include/git2/signature.h @@ -47,7 +47,7 @@ GIT_BEGIN_DECL * @offset timezone offset in minutes for the time * @return the new sig, NULl on out of memory */ -GIT_EXTERN(git_signature *) git_signature_new(const char *name, const char *email, time_t time, int offset); +GIT_EXTERN(git_signature *) git_signature_new(const char *name, const char *email, git_time_t time, int offset); /** * Create a copy of an existing signature. diff --git a/include/git2/types.h b/include/git2/types.h index 64f7fc72e..db09f3842 100644 --- a/include/git2/types.h +++ b/include/git2/types.h @@ -52,12 +52,12 @@ GIT_BEGIN_DECL #if defined(_MSC_VER) typedef __int64 git_off_t; -typedef __time64_t git_time_t; +typedef __time64_t git_time_t; #elif defined(__MINGW32__) typedef off64_t git_off_t; -typedef time_t git_time_t; +typedef time64_t git_time_t; #else /* POSIX */ @@ -66,8 +66,8 @@ typedef time_t git_time_t; * before us (directly or indirectly), they'll get 32 bit off_t in their client * app, even though /we/ define _FILE_OFFSET_BITS=64. */ -typedef long long git_off_t; -typedef time_t git_time_t; +typedef int64_t git_off_t; +typedef int64_t git_time_t; #endif @@ -129,7 +129,7 @@ typedef struct git_index git_index; /** Time in a signature */ typedef struct git_time { - time_t time; /** time in seconds from epoch */ + git_time_t time; /** time in seconds from epoch */ int offset; /** timezone offset, in minutes */ } git_time; diff --git a/src/commit.c b/src/commit.c index d5d6ebd8a..03b111da5 100644 --- a/src/commit.c +++ b/src/commit.c @@ -315,7 +315,7 @@ GIT_COMMIT_GETTER(const git_signature *, author, commit->author) GIT_COMMIT_GETTER(const git_signature *, committer, commit->committer) GIT_COMMIT_GETTER(const char *, message, commit->message) GIT_COMMIT_GETTER(const char *, message_short, commit->message_short) -GIT_COMMIT_GETTER(time_t, time, commit->committer->when.time) +GIT_COMMIT_GETTER(git_time_t, time, commit->committer->when.time) GIT_COMMIT_GETTER(int, time_offset, commit->committer->when.offset) GIT_COMMIT_GETTER(unsigned int, parentcount, commit->parent_oids.length) diff --git a/src/index.c b/src/index.c index 6a355e11b..34208731c 100644 --- a/src/index.c +++ b/src/index.c @@ -312,8 +312,8 @@ int git_index_add(git_index *index, const char *rel_path, int stage) memset(&entry, 0x0, sizeof(git_index_entry)); - entry.ctime.seconds = st.st_ctime; - entry.mtime.seconds = st.st_mtime; + entry.ctime.seconds = (git_time_t)st.st_ctime; + entry.mtime.seconds = (git_time_t)st.st_mtime; /* entry.mtime.nanoseconds = st.st_mtimensec; */ /* entry.ctime.nanoseconds = st.st_ctimensec; */ entry.dev= st.st_rdev; @@ -491,10 +491,10 @@ static size_t read_entry(git_index_entry *dest, const void *buffer, size_t buffe source = (const struct entry_short *)(buffer); - dest->ctime.seconds = (time_t)ntohl(source->ctime.seconds); - dest->ctime.nanoseconds = (time_t)ntohl(source->ctime.nanoseconds); - dest->mtime.seconds = (time_t)ntohl(source->mtime.seconds); - dest->mtime.nanoseconds = (time_t)ntohl(source->mtime.nanoseconds); + dest->ctime.seconds = (git_time_t)ntohl(source->ctime.seconds); + dest->ctime.nanoseconds = ntohl(source->ctime.nanoseconds); + dest->mtime.seconds = (git_time_t)ntohl(source->mtime.seconds); + dest->mtime.nanoseconds = ntohl(source->mtime.nanoseconds); dest->dev = ntohl(source->dev); dest->ino = ntohl(source->ino); dest->mode = ntohl(source->mode); diff --git a/src/odb_pack.c b/src/odb_pack.c index 65210f0b0..8c527bcf3 100644 --- a/src/odb_pack.c +++ b/src/odb_pack.c @@ -93,7 +93,7 @@ struct pack_file { git_oid *bad_object_sha1; /* array of git_oid */ int index_version; - time_t mtime; + git_time_t mtime; int pack_fd; unsigned pack_local:1, pack_keep:1; git_oid sha1; @@ -827,7 +827,7 @@ static int packfile_check(struct pack_file **pack_out, const char *path, int loc */ p->pack_size = (off_t)st.st_size; p->pack_local = local; - p->mtime = st.st_mtime; + p->mtime = (git_time_t)st.st_mtime; /* see if we can parse the sha1 oid in the packfile name */ if (path_len < 40 || diff --git a/src/signature.c b/src/signature.c index 13816c396..412637600 100644 --- a/src/signature.c +++ b/src/signature.c @@ -38,7 +38,7 @@ void git_signature_free(git_signature *sig) free(sig); } -git_signature *git_signature_new(const char *name, const char *email, time_t time, int offset) +git_signature *git_signature_new(const char *name, const char *email, git_time_t time, int offset) { git_signature *p = NULL; diff --git a/tests/t04-commit.c b/tests/t04-commit.c index 1140d3319..e92842435 100644 --- a/tests/t04-commit.c +++ b/tests/t04-commit.c @@ -366,7 +366,7 @@ BEGIN_TEST(details0, "query the details on a parsed commit") const git_signature *author, *committer; const char *message, *message_short; - time_t commit_time; + git_time_t commit_time; unsigned int parents, p; git_commit *parent; diff --git a/tests/t06-index.c b/tests/t06-index.c index 19b4da5c2..93ca2c04e 100644 --- a/tests/t06-index.c +++ b/tests/t06-index.c @@ -34,7 +34,7 @@ struct test_entry { unsigned int index; char path[128]; git_off_t file_size; - time_t mtime; + git_time_t mtime; }; struct test_entry TEST_ENTRIES[] = { From ae6ba7f713804e1c04febee81a8f52ea0abf443b Mon Sep 17 00:00:00 2001 From: nulltoken Date: Tue, 22 Mar 2011 19:45:01 +0100 Subject: [PATCH 39/81] Fix gitfo_mv() behavior when running on Windows When the system temporary folder is located on a different volume than the working directory into which libgit2 is executing, MoveFileEx() requires an additional flag. --- src/fileops.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/fileops.c b/src/fileops.c index 237a16a9f..a884b4002 100644 --- a/src/fileops.c +++ b/src/fileops.c @@ -263,7 +263,7 @@ int gitfo_mv(const char *from, const char *to) * file exists, the `rename` call fails. This is as * close as it gets with the Win32 API. */ - return MoveFileEx(from, to, MOVEFILE_REPLACE_EXISTING) ? GIT_SUCCESS : GIT_EOSERR; + return MoveFileEx(from, to, MOVEFILE_REPLACE_EXISTING | MOVEFILE_COPY_ALLOWED) ? GIT_SUCCESS : GIT_EOSERR; #else /* Don't even try this on Win32 */ if (!link(from, to)) { From 677a3c07f47f46dcbab4a3ee8a1e2aace6f999b0 Mon Sep 17 00:00:00 2001 From: nulltoken Date: Tue, 15 Mar 2011 22:07:01 +0100 Subject: [PATCH 40/81] Add failing test for issue 84 see https://github.com/libgit2/libgit2/issues#issue/84 --- src/fileops.c | 31 +++++++++++++++++++++++++++++++ src/fileops.h | 2 ++ tests/t12-repo.c | 22 ++++++++++++++++++++++ 3 files changed, 55 insertions(+) diff --git a/src/fileops.c b/src/fileops.c index a884b4002..4ed53bc97 100644 --- a/src/fileops.c +++ b/src/fileops.c @@ -633,3 +633,34 @@ int gitfo_cmp_path(const char *name1, int len1, int isdir1, return 0; } +static void posixify_path(char *path) +{ + while (*path) { + if (*path == '\\') + *path = '/'; + + path++; + } +} + +int gitfo_getcwd(char *buffer_out, size_t size) +{ + char *cwd_buffer; + + assert(buffer_out && size > 0); + +#ifdef GIT_WIN32 + cwd_buffer = _getcwd(buffer_out, size); +#else + cwd_buffer = getcwd(buffer_out, size); //TODO: Fixme. Ensure the required headers are correctly included +#endif + + if (cwd_buffer == NULL) + return GIT_EOSERR; + + posixify_path(buffer_out); + + git__joinpath(buffer_out, buffer_out, ""); //Ensure the path ends with a trailing slash + + return GIT_SUCCESS; +} diff --git a/src/fileops.h b/src/fileops.h index bc636fc38..bd59011d5 100644 --- a/src/fileops.h +++ b/src/fileops.h @@ -144,6 +144,8 @@ extern int gitfo_close_cached(gitfo_cache *ioc); extern int gitfo_cmp_path(const char *name1, int len1, int isdir1, const char *name2, int len2, int isdir2); +extern int gitfo_getcwd(char *buffer_out, size_t size); + /** * Clean up a provided absolute or relative directory path. * diff --git a/tests/t12-repo.c b/tests/t12-repo.c index a9a93d147..8b39e3902 100644 --- a/tests/t12-repo.c +++ b/tests/t12-repo.c @@ -162,11 +162,33 @@ BEGIN_TEST(init1, "initialize a bare repo") must_pass(ensure_repository_init(TEMP_REPO_FOLDER_NS, BARE_REPOSITORY, NULL, path_repository, NULL)); END_TEST +BEGIN_TEST(init2, "Initialize a bare repo with a relative path escaping out of the current working directory") + char path_repository[GIT_PATH_MAX]; + char current_workdir[GIT_PATH_MAX]; + const int mode = 0755; /* or 0777 ? */ + git_repository* repo; + + must_pass(gitfo_getcwd(current_workdir, sizeof(current_workdir))); + + git__joinpath(path_repository, current_workdir, "a/b/c/"); + must_pass(gitfo_mkdir_recurs(path_repository, mode)); + + must_pass(chdir(path_repository)); + + must_pass(git_repository_init(&repo, "../d/e.git", 1)); + git_repository_free(repo); + + must_pass(chdir(current_workdir)); + + git__joinpath(path_repository, current_workdir, "a/"); + must_pass(rmdir_recurs(path_repository)); +END_TEST BEGIN_SUITE(repository) ADD_TEST(odb0); ADD_TEST(odb1); ADD_TEST(init0); ADD_TEST(init1); + ADD_TEST(init2); END_SUITE From 6279abda7370ebb046571437012dc6bb96e1edba Mon Sep 17 00:00:00 2001 From: nulltoken Date: Thu, 17 Mar 2011 06:56:58 +0100 Subject: [PATCH 41/81] Make gitfo_prettify_dir_path() and gitfo_prettify_file_path() no longer externalized --- src/fileops.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/fileops.h b/src/fileops.h index bd59011d5..c2c038b26 100644 --- a/src/fileops.h +++ b/src/fileops.h @@ -168,7 +168,7 @@ extern int gitfo_getcwd(char *buffer_out, size_t size); * - GIT_SUCCESS on success; * - GIT_ERROR when the input path is invalid or escapes the current directory. */ -GIT_EXTERN(int) gitfo_prettify_dir_path(char *buffer_out, const char *path); +int gitfo_prettify_dir_path(char *buffer_out, const char *path); /** * Clean up a provided absolute or relative file path. @@ -190,6 +190,6 @@ GIT_EXTERN(int) gitfo_prettify_dir_path(char *buffer_out, const char *path); * - GIT_SUCCESS on success; * - GIT_ERROR when the input path is invalid or escapes the current directory. */ -GIT_EXTERN(int) gitfo_prettify_file_path(char *buffer_out, const char *path); +int gitfo_prettify_file_path(char *buffer_out, const char *path); #endif /* INCLUDE_fileops_h__ */ From c90292ce4f852ff9f0e8ec4ddc7cdc438745790d Mon Sep 17 00:00:00 2001 From: nulltoken Date: Fri, 18 Mar 2011 16:56:43 +0100 Subject: [PATCH 42/81] Change gitfo_prettify_dir_path() and gitfo_prettify_file_path() behavior Those functions now return prettified rooted path. --- src/fileops.c | 60 ++++++--- src/fileops.h | 7 +- src/repository.c | 10 +- tests/t00-core.c | 325 +++++++++++++++++++++++++---------------------- tests/t12-repo.c | 20 ++- 5 files changed, 230 insertions(+), 192 deletions(-) diff --git a/src/fileops.c b/src/fileops.c index 4ed53bc97..41239091b 100644 --- a/src/fileops.c +++ b/src/fileops.c @@ -443,22 +443,29 @@ int gitfo_dirent( return GIT_SUCCESS; } + +int retrieve_path_root_offset(const char *path) +{ + int offset = 0; + #ifdef GIT_WIN32 -static int is_windows_rooted_path(const char *path) -{ /* Does the root of the path look like a windows drive ? */ if (isalpha(path[0]) && (path[1] == ':')) - return GIT_SUCCESS; + offset += 2; + +#endif + + if (*(path + offset) == '/') + return offset; return GIT_ERROR; } -#endif int gitfo_mkdir_recurs(const char *path, int mode) { - int error; + int error, root_path_offset; char *pp, *sp; char *path_copy = git__strdup(path); @@ -468,12 +475,9 @@ int gitfo_mkdir_recurs(const char *path, int mode) error = GIT_SUCCESS; pp = path_copy; -#ifdef GIT_WIN32 - - if (!is_windows_rooted_path(pp)) - pp += 2; /* Skip the drive name (eg. C: or D:) */ - -#endif + root_path_offset = retrieve_path_root_offset(pp); + if (root_path_offset > 0) + pp += root_path_offset; /* On Windows, will skip the drive name (eg. C: or D:) */ while (error == GIT_SUCCESS && (sp = strchr(pp, '/')) != 0) { if (sp != pp && gitfo_isdir(path_copy) < GIT_SUCCESS) { @@ -504,6 +508,8 @@ static int retrieve_previous_path_component_start(const char *path) len = strlen(path); offset = len - 1; + //TODO: Deal with Windows rooted path + /* Skip leading slash */ if (path[start] == '/') start++; @@ -522,15 +528,25 @@ static int retrieve_previous_path_component_start(const char *path) return offset; } -int gitfo_prettify_dir_path(char *buffer_out, const char *path) +int gitfo_prettify_dir_path(char *buffer_out, size_t size, const char *path) { - int len = 0, segment_len, only_dots; + int len = 0, segment_len, only_dots, root_path_offset, error = GIT_SUCCESS; char *current; const char *buffer_out_start, *buffer_end; - buffer_out_start = buffer_out; current = (char *)path; buffer_end = path + strlen(path); + buffer_out_start = buffer_out; + + root_path_offset = retrieve_path_root_offset(path); + if (root_path_offset < 0) { + error = gitfo_getcwd(buffer_out, size); + if (error < GIT_SUCCESS) + return error; + + len = strlen(buffer_out); + buffer_out += len; + } while (current < buffer_end) { /* Prevent multiple slashes from being added to the output */ @@ -543,7 +559,7 @@ int gitfo_prettify_dir_path(char *buffer_out, const char *path) segment_len = 0; /* Copy path segment to the output */ - while (current < buffer_end && *current !='/') + while (current < buffer_end && *current != '/') { only_dots &= (*current == '.'); *buffer_out++ = *current++; @@ -568,7 +584,9 @@ int gitfo_prettify_dir_path(char *buffer_out, const char *path) *buffer_out ='\0'; len = retrieve_previous_path_component_start(buffer_out_start); - if (len < GIT_SUCCESS) + + /* Are we escaping out of the root dir? */ + if (len < 0) return GIT_EINVALIDPATH; buffer_out = (char *)buffer_out_start + len; @@ -576,7 +594,7 @@ int gitfo_prettify_dir_path(char *buffer_out, const char *path) } /* Guard against potential multiple dot path traversal (cf http://cwe.mitre.org/data/definitions/33.html) */ - if (only_dots &&segment_len > 0) + if (only_dots && segment_len > 0) return GIT_EINVALIDPATH; *buffer_out++ = '/'; @@ -588,20 +606,24 @@ int gitfo_prettify_dir_path(char *buffer_out, const char *path) return GIT_SUCCESS; } -int gitfo_prettify_file_path(char *buffer_out, const char *path) +int gitfo_prettify_file_path(char *buffer_out, size_t size, const char *path) { int error, path_len, i; const char* pattern = "/.."; path_len = strlen(path); + /* Let's make sure the filename isn't empty nor a dot */ + if (path_len == 0 || (path_len == 1 && *path == '.')) + return GIT_EINVALIDPATH; + /* Let's make sure the filename doesn't end with "/", "/." or "/.." */ for (i = 1; path_len > i && i < 4; i++) { if (!strncmp(path + path_len - i, pattern, i)) return GIT_EINVALIDPATH; } - error = gitfo_prettify_dir_path(buffer_out, path); + error = gitfo_prettify_dir_path(buffer_out, size, path); if (error < GIT_SUCCESS) return error; diff --git a/src/fileops.h b/src/fileops.h index c2c038b26..ce236f608 100644 --- a/src/fileops.h +++ b/src/fileops.h @@ -163,12 +163,13 @@ extern int gitfo_getcwd(char *buffer_out, size_t size); * the file system perspective. * * @param buffer_out buffer to populate with the normalized path. + * @param size buffer size. * @param path directory path to clean up. * @return * - GIT_SUCCESS on success; * - GIT_ERROR when the input path is invalid or escapes the current directory. */ -int gitfo_prettify_dir_path(char *buffer_out, const char *path); +int gitfo_prettify_dir_path(char *buffer_out, size_t size, const char *path); /** * Clean up a provided absolute or relative file path. @@ -185,11 +186,13 @@ int gitfo_prettify_dir_path(char *buffer_out, const char *path); * the file system perspective. * * @param buffer_out buffer to populate with the normalized path. + * @param size buffer size. * @param path file path to clean up. * @return * - GIT_SUCCESS on success; * - GIT_ERROR when the input path is invalid or escapes the current directory. */ -int gitfo_prettify_file_path(char *buffer_out, const char *path); +int gitfo_prettify_file_path(char *buffer_out, size_t size, const char *path); +int retrieve_path_root_offset(const char *path); #endif /* INCLUDE_fileops_h__ */ diff --git a/src/repository.c b/src/repository.c index 132969402..91b95a881 100644 --- a/src/repository.c +++ b/src/repository.c @@ -66,7 +66,7 @@ static int assign_repository_dirs( if (git_dir == NULL) return GIT_ENOTFOUND; - error = gitfo_prettify_dir_path(path_aux, git_dir); + error = gitfo_prettify_dir_path(path_aux, sizeof(path_aux), git_dir); if (error < GIT_SUCCESS) return error; @@ -81,7 +81,7 @@ static int assign_repository_dirs( if (git_object_directory == NULL) git__joinpath(path_aux, repo->path_repository, GIT_OBJECTS_DIR); else { - error = gitfo_prettify_dir_path(path_aux, git_object_directory); + error = gitfo_prettify_dir_path(path_aux, sizeof(path_aux), git_object_directory); if (error < GIT_SUCCESS) return error; } @@ -95,7 +95,7 @@ static int assign_repository_dirs( if (git_work_tree == NULL) repo->is_bare = 1; else { - error = gitfo_prettify_dir_path(path_aux, git_work_tree); + error = gitfo_prettify_dir_path(path_aux, sizeof(path_aux), git_work_tree); if (error < GIT_SUCCESS) return error; @@ -108,7 +108,7 @@ static int assign_repository_dirs( if (git_index_file == NULL) git__joinpath(path_aux, repo->path_repository, GIT_INDEX_FILE); else { - error = gitfo_prettify_file_path(path_aux, git_index_file); + error = gitfo_prettify_file_path(path_aux, sizeof(path_aux), git_index_file); if (error < GIT_SUCCESS) return error; } @@ -403,7 +403,7 @@ static int repo_init_find_dir(repo_init *results, const char* path) char temp_path[GIT_PATH_MAX]; int error = GIT_SUCCESS; - error = gitfo_prettify_dir_path(temp_path, path); + error = gitfo_prettify_dir_path(temp_path, sizeof(temp_path), path); if (error < GIT_SUCCESS) return error; diff --git a/tests/t00-core.c b/tests/t00-core.c index 08e42ecf4..d5fa07eb4 100644 --- a/tests/t00-core.c +++ b/tests/t00-core.c @@ -151,187 +151,204 @@ BEGIN_TEST(path2, "get the latest component in a path") #undef TOPDIR_TEST END_TEST -typedef int (normalize_path)(char *, const char *); +typedef int (normalize_path)(char *, size_t, const char *); -static int ensure_normalized(const char *input_path, const char *expected_path, normalize_path normalizer) +/* Assert flags */ +#define CWD_AS_PREFIX 1 +#define PATH_AS_SUFFIX 2 +#define ROOTED_PATH 4 + +static int ensure_normalized(const char *input_path, const char *expected_path, normalize_path normalizer, int assert_flags) { int error = GIT_SUCCESS; char buffer_out[GIT_PATH_MAX]; + char current_workdir[GIT_PATH_MAX]; - error = normalizer(buffer_out, input_path); + error = gitfo_getcwd(current_workdir, sizeof(current_workdir)); + if (error < GIT_SUCCESS) + return error; + + error = normalizer(buffer_out, sizeof(buffer_out), input_path); if (error < GIT_SUCCESS) return error; if (expected_path == NULL) return error; - if (strcmp(buffer_out, expected_path)) - error = GIT_ERROR; + if ((assert_flags & PATH_AS_SUFFIX) != 0) + if (git__suffixcmp(buffer_out, expected_path)) + return GIT_ERROR; + + if ((assert_flags & CWD_AS_PREFIX) != 0) + if (git__prefixcmp(buffer_out, current_workdir)) + return GIT_ERROR; + + if ((assert_flags & ROOTED_PATH) != 0) { + error = strcmp(expected_path, buffer_out); + } return error; } -static int ensure_dir_path_normalized(const char *input_path, const char *expected_path) +static int ensure_dir_path_normalized(const char *input_path, const char *expected_path, int assert_flags) { - return ensure_normalized(input_path, expected_path, gitfo_prettify_dir_path); + return ensure_normalized(input_path, expected_path, gitfo_prettify_dir_path, assert_flags); } -static int ensure_file_path_normalized(const char *input_path, const char *expected_path) +static int ensure_file_path_normalized(const char *input_path, const char *expected_path, int assert_flags) { - return ensure_normalized(input_path, expected_path, gitfo_prettify_file_path); + return ensure_normalized(input_path, expected_path, gitfo_prettify_file_path, assert_flags); } BEGIN_TEST(path3, "prettify and validate a path to a file") - must_pass(ensure_file_path_normalized("a", "a")); - must_pass(ensure_file_path_normalized("./testrepo.git", "testrepo.git")); - must_pass(ensure_file_path_normalized("./.git", ".git")); - must_pass(ensure_file_path_normalized("./git.", "git.")); - must_fail(ensure_file_path_normalized("git./", NULL)); - must_fail(ensure_file_path_normalized("", NULL)); - must_fail(ensure_file_path_normalized(".", NULL)); - must_fail(ensure_file_path_normalized("./", NULL)); - must_fail(ensure_file_path_normalized("./.", NULL)); - must_fail(ensure_file_path_normalized("./..", NULL)); - must_fail(ensure_file_path_normalized("../.", NULL)); - must_fail(ensure_file_path_normalized("./.././/", NULL)); - must_fail(ensure_file_path_normalized("dir/..", NULL)); - must_fail(ensure_file_path_normalized("dir/sub/../..", NULL)); - must_fail(ensure_file_path_normalized("dir/sub/..///..", NULL)); - must_fail(ensure_file_path_normalized("dir/sub///../..", NULL)); - must_fail(ensure_file_path_normalized("dir/sub///..///..", NULL)); - must_fail(ensure_file_path_normalized("dir/sub/../../..", NULL)); - must_pass(ensure_file_path_normalized("dir", "dir")); - must_fail(ensure_file_path_normalized("dir//", NULL)); - must_pass(ensure_file_path_normalized("./dir", "dir")); - must_fail(ensure_file_path_normalized("dir/.", NULL)); - must_fail(ensure_file_path_normalized("dir///./", NULL)); - must_fail(ensure_file_path_normalized("dir/sub/..", NULL)); - must_fail(ensure_file_path_normalized("dir//sub/..",NULL)); - must_fail(ensure_file_path_normalized("dir//sub/../", NULL)); - must_fail(ensure_file_path_normalized("dir/sub/../", NULL)); - must_fail(ensure_file_path_normalized("dir/sub/../.", NULL)); - must_fail(ensure_file_path_normalized("dir/s1/../s2/", NULL)); - must_fail(ensure_file_path_normalized("d1/s1///s2/..//../s3/", NULL)); - must_pass(ensure_file_path_normalized("d1/s1//../s2/../../d2", "d2")); - must_fail(ensure_file_path_normalized("dir/sub/../", NULL)); - must_fail(ensure_file_path_normalized("....", NULL)); - must_fail(ensure_file_path_normalized("...", NULL)); - must_fail(ensure_file_path_normalized("./...", NULL)); - must_fail(ensure_file_path_normalized("d1/...", NULL)); - must_fail(ensure_file_path_normalized("d1/.../", NULL)); - must_fail(ensure_file_path_normalized("d1/.../d2", NULL)); + must_pass(ensure_file_path_normalized("a", "a", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_file_path_normalized("./testrepo.git", "testrepo.git", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_file_path_normalized("./.git", ".git", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_file_path_normalized("./git.", "git.", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_fail(ensure_file_path_normalized("git./", NULL, 0)); + must_fail(ensure_file_path_normalized("", NULL, 0)); + must_fail(ensure_file_path_normalized(".", NULL, 0)); + must_fail(ensure_file_path_normalized("./", NULL, 0)); + must_fail(ensure_file_path_normalized("./.", NULL, 0)); + must_fail(ensure_file_path_normalized("./..", NULL, 0)); + must_fail(ensure_file_path_normalized("../.", NULL, 0)); + must_fail(ensure_file_path_normalized("./.././/", NULL, 0)); + must_fail(ensure_file_path_normalized("dir/..", NULL, 0)); + must_fail(ensure_file_path_normalized("dir/sub/../..", NULL, 0)); + must_fail(ensure_file_path_normalized("dir/sub/..///..", NULL, 0)); + must_fail(ensure_file_path_normalized("dir/sub///../..", NULL, 0)); + must_fail(ensure_file_path_normalized("dir/sub///..///..", NULL, 0)); + must_fail(ensure_file_path_normalized("dir/sub/../../..", NULL, 0)); + must_pass(ensure_file_path_normalized("dir", "dir", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_fail(ensure_file_path_normalized("dir//", NULL, 0)); + must_pass(ensure_file_path_normalized("./dir", "dir", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_fail(ensure_file_path_normalized("dir/.", NULL, 0)); + must_fail(ensure_file_path_normalized("dir///./", NULL, 0)); + must_fail(ensure_file_path_normalized("dir/sub/..", NULL, 0)); + must_fail(ensure_file_path_normalized("dir//sub/..",NULL, 0)); + must_fail(ensure_file_path_normalized("dir//sub/../", NULL, 0)); + must_fail(ensure_file_path_normalized("dir/sub/../", NULL, 0)); + must_fail(ensure_file_path_normalized("dir/sub/../.", NULL, 0)); + must_fail(ensure_file_path_normalized("dir/s1/../s2/", NULL, 0)); + must_fail(ensure_file_path_normalized("d1/s1///s2/..//../s3/", NULL, 0)); + must_pass(ensure_file_path_normalized("d1/s1//../s2/../../d2", "d2", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_fail(ensure_file_path_normalized("dir/sub/../", NULL, 0)); + must_pass(ensure_file_path_normalized("../../a/../../b/c/d/../../e", "b/e", PATH_AS_SUFFIX)); + must_fail(ensure_file_path_normalized("....", NULL, 0)); + must_fail(ensure_file_path_normalized("...", NULL, 0)); + must_fail(ensure_file_path_normalized("./...", NULL, 0)); + must_fail(ensure_file_path_normalized("d1/...", NULL, 0)); + must_fail(ensure_file_path_normalized("d1/.../", NULL, 0)); + must_fail(ensure_file_path_normalized("d1/.../d2", NULL, 0)); - must_pass(ensure_file_path_normalized("/a", "/a")); - must_pass(ensure_file_path_normalized("/./testrepo.git", "/testrepo.git")); - must_pass(ensure_file_path_normalized("/./.git", "/.git")); - must_pass(ensure_file_path_normalized("/./git.", "/git.")); - must_fail(ensure_file_path_normalized("/git./", NULL)); - must_fail(ensure_file_path_normalized("/", NULL)); - must_fail(ensure_file_path_normalized("/.", NULL)); - must_fail(ensure_file_path_normalized("/./", NULL)); - must_fail(ensure_file_path_normalized("/./.", NULL)); - must_fail(ensure_file_path_normalized("/./..", NULL)); - must_fail(ensure_file_path_normalized("/../.", NULL)); - must_fail(ensure_file_path_normalized("/./.././/", NULL)); - must_fail(ensure_file_path_normalized("/dir/..", NULL)); - must_fail(ensure_file_path_normalized("/dir/sub/../..", NULL)); - must_fail(ensure_file_path_normalized("/dir/sub/..///..", NULL)); - must_fail(ensure_file_path_normalized("/dir/sub///../..", NULL)); - must_fail(ensure_file_path_normalized("/dir/sub///..///..", NULL)); - must_fail(ensure_file_path_normalized("/dir/sub/../../..", NULL)); - must_pass(ensure_file_path_normalized("/dir", "/dir")); - must_fail(ensure_file_path_normalized("/dir//", NULL)); - must_pass(ensure_file_path_normalized("/./dir", "/dir")); - must_fail(ensure_file_path_normalized("/dir/.", NULL)); - must_fail(ensure_file_path_normalized("/dir///./", NULL)); - must_fail(ensure_file_path_normalized("/dir/sub/..", NULL)); - must_fail(ensure_file_path_normalized("/dir//sub/..",NULL)); - must_fail(ensure_file_path_normalized("/dir//sub/../", NULL)); - must_fail(ensure_file_path_normalized("/dir/sub/../", NULL)); - must_fail(ensure_file_path_normalized("/dir/sub/../.", NULL)); - must_fail(ensure_file_path_normalized("/dir/s1/../s2/", NULL)); - must_fail(ensure_file_path_normalized("/d1/s1///s2/..//../s3/", NULL)); - must_pass(ensure_file_path_normalized("/d1/s1//../s2/../../d2", "/d2")); - must_fail(ensure_file_path_normalized("/dir/sub/../", NULL)); - must_fail(ensure_file_path_normalized("/....", NULL)); - must_fail(ensure_file_path_normalized("/...", NULL)); - must_fail(ensure_file_path_normalized("/./...", NULL)); - must_fail(ensure_file_path_normalized("/d1/...", NULL)); - must_fail(ensure_file_path_normalized("/d1/.../", NULL)); - must_fail(ensure_file_path_normalized("/d1/.../d2", NULL)); + must_pass(ensure_file_path_normalized("/a", "/a", ROOTED_PATH)); + must_pass(ensure_file_path_normalized("/./testrepo.git", "/testrepo.git", ROOTED_PATH)); + must_pass(ensure_file_path_normalized("/./.git", "/.git", ROOTED_PATH)); + must_pass(ensure_file_path_normalized("/./git.", "/git.", ROOTED_PATH)); + must_fail(ensure_file_path_normalized("/git./", NULL, 0)); + must_fail(ensure_file_path_normalized("/", NULL, 0)); + must_fail(ensure_file_path_normalized("/.", NULL, 0)); + must_fail(ensure_file_path_normalized("/./", NULL, 0)); + must_fail(ensure_file_path_normalized("/./.", NULL, 0)); + must_fail(ensure_file_path_normalized("/./..", NULL, 0)); + must_fail(ensure_file_path_normalized("/../.", NULL, 0)); + must_fail(ensure_file_path_normalized("/./.././/", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir/..", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir/sub/../..", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir/sub/..///..", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir/sub///../..", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir/sub///..///..", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir/sub/../../..", NULL, 0)); + must_pass(ensure_file_path_normalized("/dir", "/dir", 0)); + must_fail(ensure_file_path_normalized("/dir//", NULL, 0)); + must_pass(ensure_file_path_normalized("/./dir", "/dir", 0)); + must_fail(ensure_file_path_normalized("/dir/.", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir///./", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir/sub/..", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir//sub/..",NULL, 0)); + must_fail(ensure_file_path_normalized("/dir//sub/../", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir/sub/../", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir/sub/../.", NULL, 0)); + must_fail(ensure_file_path_normalized("/dir/s1/../s2/", NULL, 0)); + must_fail(ensure_file_path_normalized("/d1/s1///s2/..//../s3/", NULL, 0)); + must_pass(ensure_file_path_normalized("/d1/s1//../s2/../../d2", "/d2", 0)); + must_fail(ensure_file_path_normalized("/dir/sub/../", NULL, 0)); + must_fail(ensure_file_path_normalized("/....", NULL, 0)); + must_fail(ensure_file_path_normalized("/...", NULL, 0)); + must_fail(ensure_file_path_normalized("/./...", NULL, 0)); + must_fail(ensure_file_path_normalized("/d1/...", NULL, 0)); + must_fail(ensure_file_path_normalized("/d1/.../", NULL, 0)); + must_fail(ensure_file_path_normalized("/d1/.../d2", NULL, 0)); END_TEST BEGIN_TEST(path4, "validate and prettify a path to a folder") - must_pass(ensure_dir_path_normalized("./testrepo.git", "testrepo.git/")); - must_pass(ensure_dir_path_normalized("./.git", ".git/")); - must_pass(ensure_dir_path_normalized("./git.", "git./")); - must_pass(ensure_dir_path_normalized("git./", "git./")); - must_pass(ensure_dir_path_normalized("", "")); - must_pass(ensure_dir_path_normalized(".", "")); - must_pass(ensure_dir_path_normalized("./", "")); - must_pass(ensure_dir_path_normalized("./.", "")); - must_fail(ensure_dir_path_normalized("./..", NULL)); - must_fail(ensure_dir_path_normalized("../.", NULL)); - must_fail(ensure_dir_path_normalized("./.././/", NULL)); - must_pass(ensure_dir_path_normalized("dir/..", "")); - must_pass(ensure_dir_path_normalized("dir/sub/../..", "")); - must_pass(ensure_dir_path_normalized("dir/sub/..///..", "")); - must_pass(ensure_dir_path_normalized("dir/sub///../..", "")); - must_pass(ensure_dir_path_normalized("dir/sub///..///..", "")); - must_fail(ensure_dir_path_normalized("dir/sub/../../..", NULL)); - must_pass(ensure_dir_path_normalized("dir", "dir/")); - must_pass(ensure_dir_path_normalized("dir//", "dir/")); - must_pass(ensure_dir_path_normalized("./dir", "dir/")); - must_pass(ensure_dir_path_normalized("dir/.", "dir/")); - must_pass(ensure_dir_path_normalized("dir///./", "dir/")); - must_pass(ensure_dir_path_normalized("dir/sub/..", "dir/")); - must_pass(ensure_dir_path_normalized("dir//sub/..", "dir/")); - must_pass(ensure_dir_path_normalized("dir//sub/../", "dir/")); - must_pass(ensure_dir_path_normalized("dir/sub/../", "dir/")); - must_pass(ensure_dir_path_normalized("dir/sub/../.", "dir/")); - must_pass(ensure_dir_path_normalized("dir/s1/../s2/", "dir/s2/")); - must_pass(ensure_dir_path_normalized("d1/s1///s2/..//../s3/", "d1/s3/")); - must_pass(ensure_dir_path_normalized("d1/s1//../s2/../../d2", "d2/")); - must_pass(ensure_dir_path_normalized("dir/sub/../", "dir/")); - must_fail(ensure_dir_path_normalized("....", NULL)); - must_fail(ensure_dir_path_normalized("...", NULL)); - must_fail(ensure_dir_path_normalized("./...", NULL)); - must_fail(ensure_dir_path_normalized("d1/...", NULL)); - must_fail(ensure_dir_path_normalized("d1/.../", NULL)); - must_fail(ensure_dir_path_normalized("d1/.../d2", NULL)); + must_pass(ensure_dir_path_normalized("./testrepo.git", "testrepo.git/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("./.git", ".git/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("./git.", "git./", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("git./", "git./", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("", "", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized(".", "", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("./", "", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("./.", "", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/..", "", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/sub/../..", "", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/sub/..///..", "", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/sub///../..", "", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/sub///..///..", "", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir//", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("./dir", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/.", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir///./", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/sub/..", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir//sub/..", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir//sub/../", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/sub/../", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/sub/../.", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/s1/../s2/", "dir/s2/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("d1/s1///s2/..//../s3/", "d1/s3/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("d1/s1//../s2/../../d2", "d2/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("dir/sub/../", "dir/", CWD_AS_PREFIX | PATH_AS_SUFFIX)); + must_pass(ensure_dir_path_normalized("../../a/../../b/c/d/../../e", "b/e/", PATH_AS_SUFFIX)); + must_fail(ensure_dir_path_normalized("....", NULL, 0)); + must_fail(ensure_dir_path_normalized("...", NULL, 0)); + must_fail(ensure_dir_path_normalized("./...", NULL, 0)); + must_fail(ensure_dir_path_normalized("d1/...", NULL, 0)); + must_fail(ensure_dir_path_normalized("d1/.../", NULL, 0)); + must_fail(ensure_dir_path_normalized("d1/.../d2", NULL, 0)); - must_pass(ensure_dir_path_normalized("/./testrepo.git", "/testrepo.git/")); - must_pass(ensure_dir_path_normalized("/./.git", "/.git/")); - must_pass(ensure_dir_path_normalized("/./git.", "/git./")); - must_pass(ensure_dir_path_normalized("/git./", "/git./")); - must_pass(ensure_dir_path_normalized("/", "/")); - must_pass(ensure_dir_path_normalized("//", "/")); - must_pass(ensure_dir_path_normalized("///", "/")); - must_pass(ensure_dir_path_normalized("/.", "/")); - must_pass(ensure_dir_path_normalized("/./", "/")); - must_fail(ensure_dir_path_normalized("/./..", NULL)); - must_fail(ensure_dir_path_normalized("/../.", NULL)); - must_fail(ensure_dir_path_normalized("/./.././/", NULL)); - must_pass(ensure_dir_path_normalized("/dir/..", "/")); - must_pass(ensure_dir_path_normalized("/dir/sub/../..", "/")); - must_fail(ensure_dir_path_normalized("/dir/sub/../../..", NULL)); - must_pass(ensure_dir_path_normalized("/dir", "/dir/")); - must_pass(ensure_dir_path_normalized("/dir//", "/dir/")); - must_pass(ensure_dir_path_normalized("/./dir", "/dir/")); - must_pass(ensure_dir_path_normalized("/dir/.", "/dir/")); - must_pass(ensure_dir_path_normalized("/dir///./", "/dir/")); - must_pass(ensure_dir_path_normalized("/dir//sub/..", "/dir/")); - must_pass(ensure_dir_path_normalized("/dir/sub/../", "/dir/")); - must_pass(ensure_dir_path_normalized("//dir/sub/../.", "/dir/")); - must_pass(ensure_dir_path_normalized("/dir/s1/../s2/", "/dir/s2/")); - must_pass(ensure_dir_path_normalized("/d1/s1///s2/..//../s3/", "/d1/s3/")); - must_pass(ensure_dir_path_normalized("/d1/s1//../s2/../../d2", "/d2/")); - must_fail(ensure_dir_path_normalized("/....", NULL)); - must_fail(ensure_dir_path_normalized("/...", NULL)); - must_fail(ensure_dir_path_normalized("/./...", NULL)); - must_fail(ensure_dir_path_normalized("/d1/...", NULL)); - must_fail(ensure_dir_path_normalized("/d1/.../", NULL)); - must_fail(ensure_dir_path_normalized("/d1/.../d2", NULL)); + must_pass(ensure_dir_path_normalized("/./testrepo.git", "/testrepo.git/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/./.git", "/.git/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/./git.", "/git./", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/git./", "/git./", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/", "/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("//", "/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("///", "/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/.", "/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/./", "/", ROOTED_PATH)); + must_fail(ensure_dir_path_normalized("/./..", NULL, 0)); + must_fail(ensure_dir_path_normalized("/../.", NULL, 0)); + must_fail(ensure_dir_path_normalized("/./.././/", NULL, 0)); + must_pass(ensure_dir_path_normalized("/dir/..", "/", 0)); + must_pass(ensure_dir_path_normalized("/dir/sub/../..", "/", 0)); + must_fail(ensure_dir_path_normalized("/dir/sub/../../..", NULL, 0)); + must_pass(ensure_dir_path_normalized("/dir", "/dir/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/dir//", "/dir/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/./dir", "/dir/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/dir/.", "/dir/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/dir///./", "/dir/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/dir//sub/..", "/dir/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/dir/sub/../", "/dir/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("//dir/sub/../.", "/dir/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/dir/s1/../s2/", "/dir/s2/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/d1/s1///s2/..//../s3/", "/d1/s3/", ROOTED_PATH)); + must_pass(ensure_dir_path_normalized("/d1/s1//../s2/../../d2", "/d2/", ROOTED_PATH)); + must_fail(ensure_dir_path_normalized("/....", NULL, 0)); + must_fail(ensure_dir_path_normalized("/...", NULL, 0)); + must_fail(ensure_dir_path_normalized("/./...", NULL, 0)); + must_fail(ensure_dir_path_normalized("/d1/...", NULL, 0)); + must_fail(ensure_dir_path_normalized("/d1/.../", NULL, 0)); + must_fail(ensure_dir_path_normalized("/d1/.../d2", NULL, 0)); END_TEST static int ensure_joinpath(const char *path_a, const char *path_b, const char *expected_path) diff --git a/tests/t12-repo.c b/tests/t12-repo.c index 8b39e3902..2c7f1316a 100644 --- a/tests/t12-repo.c +++ b/tests/t12-repo.c @@ -113,22 +113,18 @@ static int ensure_repository_init( return GIT_ERROR; if (repo->path_workdir != NULL || expected_working_directory != NULL) { - if (strcmp(repo->path_workdir, expected_working_directory) != 0) - //return GIT_ERROR; + if (git__suffixcmp(repo->path_workdir, expected_working_directory) != 0) goto cleanup; } - if (strcmp(repo->path_odb, path_odb) != 0) - //return GIT_ERROR; + if (git__suffixcmp(repo->path_odb, path_odb) != 0) goto cleanup; - if (strcmp(repo->path_repository, expected_path_repository) != 0) - //return GIT_ERROR; + if (git__suffixcmp(repo->path_repository, expected_path_repository) != 0) goto cleanup; if (repo->path_index != NULL || expected_path_index != NULL) { - if (strcmp(repo->path_index, expected_path_index) != 0) - //return GIT_ERROR; + if (git__suffixcmp(repo->path_index, expected_path_index) != 0) goto cleanup; } @@ -170,18 +166,18 @@ BEGIN_TEST(init2, "Initialize a bare repo with a relative path escaping out of t must_pass(gitfo_getcwd(current_workdir, sizeof(current_workdir))); - git__joinpath(path_repository, current_workdir, "a/b/c/"); + git__joinpath(path_repository, TEMP_REPO_FOLDER, "a/b/c/"); must_pass(gitfo_mkdir_recurs(path_repository, mode)); must_pass(chdir(path_repository)); must_pass(git_repository_init(&repo, "../d/e.git", 1)); + must_pass(git__suffixcmp(repo->path_repository, "/a/b/d/e.git/")); + git_repository_free(repo); must_pass(chdir(current_workdir)); - - git__joinpath(path_repository, current_workdir, "a/"); - must_pass(rmdir_recurs(path_repository)); + rmdir_recurs(TEMP_REPO_FOLDER); END_TEST BEGIN_SUITE(repository) From 3644e98fd984c039d0c84b2311b410583739b6a5 Mon Sep 17 00:00:00 2001 From: nulltoken Date: Fri, 18 Mar 2011 19:10:36 +0100 Subject: [PATCH 43/81] Fix detection of attempt to escape the root directory on Windows --- src/fileops.c | 12 +++++++----- tests/t00-core.c | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+), 5 deletions(-) diff --git a/src/fileops.c b/src/fileops.c index 41239091b..e440ea6ab 100644 --- a/src/fileops.c +++ b/src/fileops.c @@ -503,13 +503,15 @@ int gitfo_mkdir_recurs(const char *path, int mode) static int retrieve_previous_path_component_start(const char *path) { - int offset, len, start = 0; - + int offset, len, root_offset, start = 0; + + root_offset = retrieve_path_root_offset(path); + if (root_offset > -1) + start += root_offset; + len = strlen(path); offset = len - 1; - //TODO: Deal with Windows rooted path - /* Skip leading slash */ if (path[start] == '/') start++; @@ -518,7 +520,7 @@ static int retrieve_previous_path_component_start(const char *path) if (path[offset] == '/') offset--; - if (offset < 0) + if (offset < root_offset) return GIT_ERROR; while (offset > start && path[offset-1] != '/') { diff --git a/tests/t00-core.c b/tests/t00-core.c index d5fa07eb4..4cb111428 100644 --- a/tests/t00-core.c +++ b/tests/t00-core.c @@ -389,6 +389,37 @@ BEGIN_TEST(path6, "properly join path components for more than one path") must_pass(ensure_joinpath_n("a", "b", "", "/c/d", "a/b/c/d")); END_TEST +static int count_number_of_path_segments(const char *path) +{ + int number = 0; + char *current = (char *)path; + + while (*current) + { + if (*current++ == '/') + number++; + } + + assert (number > 0); + + return --number; +} + +BEGIN_TEST(path7, "prevent a path which escapes the root directory from being prettified") + char current_workdir[GIT_PATH_MAX]; + char prettified[GIT_PATH_MAX]; + int i = 0, number_to_escape; + + must_pass(gitfo_getcwd(current_workdir, sizeof(current_workdir))); + + number_to_escape = count_number_of_path_segments(current_workdir); + + for (i = 0; i < number_to_escape + 1; i++) + git__joinpath(current_workdir, current_workdir, "../"); + + must_fail(gitfo_prettify_dir_path(prettified, sizeof(prettified), current_workdir)); +END_TEST + typedef struct name_data { int count; /* return count */ char *name; /* filename */ @@ -645,6 +676,7 @@ BEGIN_SUITE(core) ADD_TEST(path4); ADD_TEST(path5); ADD_TEST(path6); + ADD_TEST(path7); ADD_TEST(dirent0); ADD_TEST(dirent1); From ba1bdf86e74777f1a597ac2737dddd7965f41711 Mon Sep 17 00:00:00 2001 From: nulltoken Date: Fri, 18 Mar 2011 22:27:15 +0100 Subject: [PATCH 44/81] Improve test coverage of new path prettifying behavior --- tests/t12-repo.c | 35 ++++++++++++++++++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/tests/t12-repo.c b/tests/t12-repo.c index 2c7f1316a..9b8933f54 100644 --- a/tests/t12-repo.c +++ b/tests/t12-repo.c @@ -158,7 +158,7 @@ BEGIN_TEST(init1, "initialize a bare repo") must_pass(ensure_repository_init(TEMP_REPO_FOLDER_NS, BARE_REPOSITORY, NULL, path_repository, NULL)); END_TEST -BEGIN_TEST(init2, "Initialize a bare repo with a relative path escaping out of the current working directory") +BEGIN_TEST(init2, "Initialize and open a bare repo with a relative path escaping out of the current working directory") char path_repository[GIT_PATH_MAX]; char current_workdir[GIT_PATH_MAX]; const int mode = 0755; /* or 0777 ? */ @@ -176,6 +176,38 @@ BEGIN_TEST(init2, "Initialize a bare repo with a relative path escaping out of t git_repository_free(repo); + must_pass(git_repository_open(&repo, "../d/e.git")); + + git_repository_free(repo); + + must_pass(chdir(current_workdir)); + rmdir_recurs(TEMP_REPO_FOLDER); +END_TEST + +BEGIN_TEST(open2, "Open a bare repository with a relative path escaping out of the current working directory") + char new_current_workdir[GIT_PATH_MAX]; + char current_workdir[GIT_PATH_MAX]; + char path_repository[GIT_PATH_MAX]; + + const int mode = 0755; /* or 0777 ? */ + git_repository* repo; + + /* Setup the repository to open */ + must_pass(gitfo_getcwd(current_workdir, sizeof(current_workdir))); + strcpy(path_repository, current_workdir); + git__joinpath_n(path_repository, 3, path_repository, TEMP_REPO_FOLDER, "a/d/e.git"); + must_pass(gitfo_mkdir_recurs(path_repository, mode)); + must_pass(copydir_recurs(REPOSITORY_FOLDER, path_repository)); + + /* Change the current working directory */ + git__joinpath(new_current_workdir, TEMP_REPO_FOLDER, "a/b/c/"); + must_pass(gitfo_mkdir_recurs(new_current_workdir, mode)); + must_pass(chdir(new_current_workdir)); + + must_pass(git_repository_open(&repo, "../../d/e.git")); + + git_repository_free(repo); + must_pass(chdir(current_workdir)); rmdir_recurs(TEMP_REPO_FOLDER); END_TEST @@ -186,5 +218,6 @@ BEGIN_SUITE(repository) ADD_TEST(init0); ADD_TEST(init1); ADD_TEST(init2); + ADD_TEST(open2); END_SUITE From f428ae615b4386e78cb337386b3d8e190e921766 Mon Sep 17 00:00:00 2001 From: nulltoken Date: Sat, 19 Mar 2011 08:19:34 +0100 Subject: [PATCH 45/81] Slightly enforce copy_recurs() behavior The folder creation is now decorrelated from the recursive parsing of the source tree structure. --- tests/t12-repo.c | 1 - tests/test_helpers.c | 9 ++------- 2 files changed, 2 insertions(+), 8 deletions(-) diff --git a/tests/t12-repo.c b/tests/t12-repo.c index 9b8933f54..38e350958 100644 --- a/tests/t12-repo.c +++ b/tests/t12-repo.c @@ -196,7 +196,6 @@ BEGIN_TEST(open2, "Open a bare repository with a relative path escaping out of t must_pass(gitfo_getcwd(current_workdir, sizeof(current_workdir))); strcpy(path_repository, current_workdir); git__joinpath_n(path_repository, 3, path_repository, TEMP_REPO_FOLDER, "a/d/e.git"); - must_pass(gitfo_mkdir_recurs(path_repository, mode)); must_pass(copydir_recurs(REPOSITORY_FOLDER, path_repository)); /* Change the current working directory */ diff --git a/tests/test_helpers.c b/tests/test_helpers.c index 588461135..1d345356a 100644 --- a/tests/test_helpers.c +++ b/tests/test_helpers.c @@ -141,7 +141,7 @@ int copy_file(const char *src, const char *dst) if (gitfo_read_file(&source_buf, src) < GIT_SUCCESS) return GIT_ENOTFOUND; - dst_fd = gitfo_creat(dst, 0644); + dst_fd = gitfo_creat_force(dst, 0644); if (dst_fd < 0) goto cleanup; @@ -211,18 +211,13 @@ typedef struct { static int copy_filesystem_element_recurs(void *_data, char *source) { - const int mode = 0755; /* or 0777 ? */ copydir_data *data = (copydir_data *)_data; data->dst[data->dst_len] = 0; git__joinpath(data->dst, data->dst, source + data->src_len); - if (gitfo_isdir(source) == GIT_SUCCESS) { - if (gitfo_mkdir(data->dst, mode) < GIT_SUCCESS) - return GIT_EOSERR; - + if (gitfo_isdir(source) == GIT_SUCCESS) return gitfo_dirent(source, GIT_PATH_MAX, copy_filesystem_element_recurs, _data); - } return copy_file(source, data->dst); } From 2b861d7589467734c2371c4daaab6e540857b988 Mon Sep 17 00:00:00 2001 From: nulltoken Date: Sat, 19 Mar 2011 08:31:40 +0100 Subject: [PATCH 46/81] Add detection of incorrect usage to git__joinpath() --- src/util.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/util.c b/src/util.c index bfc4f7b27..995daf314 100644 --- a/src/util.c +++ b/src/util.c @@ -223,6 +223,9 @@ void git__joinpath_n(char *buffer_out, int count, ...) int len; path = va_arg(ap, const char *); + + assert((i == 0) || path != buffer_start); + if (i > 0 && *path == '/' && buffer_out > buffer_start && buffer_out[-1] == '/') path++; From 3fe9c60c6ad66d4637c7669c0661b5097918ddbd Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 23 Mar 2011 00:21:30 +0200 Subject: [PATCH 47/81] Add empty bare and normal repos to test resources We have removed the hooks folder because it takes a lot of space --- tests/resources/empty_bare.git/HEAD | Bin 0 -> 23 bytes tests/resources/empty_bare.git/config | Bin 0 -> 131 bytes tests/resources/empty_bare.git/description | Bin 0 -> 73 bytes tests/resources/empty_bare.git/info/exclude | Bin 0 -> 240 bytes .../empty_bare.git/objects/info/dummy-marker.txt | Bin .../empty_bare.git/objects/pack/dummy-marker.txt | Bin .../empty_bare.git/refs/heads/dummy-marker.txt | Bin .../empty_bare.git/refs/tags/dummy-marker.txt | Bin tests/resources/empty_standard_repo/.gitted/HEAD | Bin 0 -> 23 bytes .../resources/empty_standard_repo/.gitted/config | Bin 0 -> 157 bytes .../empty_standard_repo/.gitted/description | Bin 0 -> 73 bytes .../empty_standard_repo/.gitted/info/exclude | Bin 0 -> 240 bytes .../.gitted/objects/info/dummy-marker.txt | Bin .../.gitted/objects/pack/dummy-marker.txt | Bin .../.gitted/refs/heads/dummy-marker.txt | Bin .../.gitted/refs/tags/dummy-marker.txt | Bin 16 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 tests/resources/empty_bare.git/HEAD create mode 100644 tests/resources/empty_bare.git/config create mode 100644 tests/resources/empty_bare.git/description create mode 100644 tests/resources/empty_bare.git/info/exclude create mode 100644 tests/resources/empty_bare.git/objects/info/dummy-marker.txt create mode 100644 tests/resources/empty_bare.git/objects/pack/dummy-marker.txt create mode 100644 tests/resources/empty_bare.git/refs/heads/dummy-marker.txt create mode 100644 tests/resources/empty_bare.git/refs/tags/dummy-marker.txt create mode 100644 tests/resources/empty_standard_repo/.gitted/HEAD create mode 100644 tests/resources/empty_standard_repo/.gitted/config create mode 100644 tests/resources/empty_standard_repo/.gitted/description create mode 100644 tests/resources/empty_standard_repo/.gitted/info/exclude create mode 100644 tests/resources/empty_standard_repo/.gitted/objects/info/dummy-marker.txt create mode 100644 tests/resources/empty_standard_repo/.gitted/objects/pack/dummy-marker.txt create mode 100644 tests/resources/empty_standard_repo/.gitted/refs/heads/dummy-marker.txt create mode 100644 tests/resources/empty_standard_repo/.gitted/refs/tags/dummy-marker.txt diff --git a/tests/resources/empty_bare.git/HEAD b/tests/resources/empty_bare.git/HEAD new file mode 100644 index 0000000000000000000000000000000000000000..cb089cd89a7d7686d284d8761201649346b5aa1c GIT binary patch literal 23 ecmXR)O|w!cN=+-)&qz&7Db~+TEG|hc;sO9;xClW2 literal 0 HcmV?d00001 diff --git a/tests/resources/empty_bare.git/config b/tests/resources/empty_bare.git/config new file mode 100644 index 0000000000000000000000000000000000000000..90e16477bddfa48ec34154f9d7211aac4e32a511 GIT binary patch literal 131 zcmYL>F%Ezr3;<_%g-`GUZbnBRU>s12HUc$mHR1Q-z literal 0 HcmV?d00001 diff --git a/tests/resources/empty_bare.git/description b/tests/resources/empty_bare.git/description new file mode 100644 index 0000000000000000000000000000000000000000..498b267a8c7812490d6479839c5577eaaec79d62 GIT binary patch literal 73 zcmWH|%S+5nO;IRHEyyp$t+PQ$;d2LNXyJgRZve!Elw`VEGWs$&r??@ Q$yWgB0LrH#Y0~2Y0PnOK(EtDd literal 0 HcmV?d00001 diff --git a/tests/resources/empty_bare.git/info/exclude b/tests/resources/empty_bare.git/info/exclude new file mode 100644 index 0000000000000000000000000000000000000000..a5196d1be8fb59edf8062bef36d3a602e0812139 GIT binary patch literal 240 zcmXYqK?=e!6h!x)VxWtv*mf_t5?px$aS_{}Hj?C*}oMPd`mF*(s_9TLV@qN9ge_KWI9)sd8MlRa2 zXaRqt8cCf_(;P!4#v%zjWnjY?9-x23vH`PVKK2BT);-swYt{-=eU0H4(pLN73O4eu U^rVDm@k>7$ViLcSFO^&T0N7wQ1ONa4 literal 0 HcmV?d00001 diff --git a/tests/resources/empty_standard_repo/.gitted/description b/tests/resources/empty_standard_repo/.gitted/description new file mode 100644 index 0000000000000000000000000000000000000000..498b267a8c7812490d6479839c5577eaaec79d62 GIT binary patch literal 73 zcmWH|%S+5nO;IRHEyyp$t+PQ$;d2LNXyJgRZve!Elw`VEGWs$&r??@ Q$yWgB0LrH#Y0~2Y0PnOK(EtDd literal 0 HcmV?d00001 diff --git a/tests/resources/empty_standard_repo/.gitted/info/exclude b/tests/resources/empty_standard_repo/.gitted/info/exclude new file mode 100644 index 0000000000000000000000000000000000000000..a5196d1be8fb59edf8062bef36d3a602e0812139 GIT binary patch literal 240 zcmXYqK?=e!6h!x)VxWtv*mf_t5?px$aS_{}Hj?C* Date: Sat, 5 Mar 2011 14:26:22 +0100 Subject: [PATCH 48/81] Add remove_placeholders() test helper function which recursively removes marker files from a directory structure --- tests/test_helpers.c | 28 ++++++++++++++++++++++++++++ tests/test_helpers.h | 1 + 2 files changed, 29 insertions(+) diff --git a/tests/test_helpers.c b/tests/test_helpers.c index 1d345356a..760de238b 100644 --- a/tests/test_helpers.c +++ b/tests/test_helpers.c @@ -256,3 +256,31 @@ void close_temp_repo(git_repository *repo) git_repository_free(repo); rmdir_recurs(TEMP_REPO_FOLDER); } + +static int remove_placeholders_recurs(void *filename, char *path) +{ + char passed_filename[GIT_PATH_MAX]; + char *data = (char *)filename; + + if (!gitfo_isdir(path)) + return gitfo_dirent(path, GIT_PATH_MAX, remove_placeholders_recurs, data); + + if (git__basename_r(passed_filename, sizeof(passed_filename), path) < GIT_SUCCESS) + return GIT_EINVALIDPATH; + + if (!strcmp(data, passed_filename)) + return gitfo_unlink(path); + + return GIT_SUCCESS; +} + +int remove_placeholders(char *directory_path, char *filename) +{ + char buffer[GIT_PATH_MAX]; + + if (gitfo_isdir(directory_path)) + return GIT_EINVALIDPATH; + + strcpy(buffer, directory_path); + return remove_placeholders_recurs(filename, buffer); +} diff --git a/tests/test_helpers.h b/tests/test_helpers.h index 9a24ebccf..19c8ae55c 100644 --- a/tests/test_helpers.h +++ b/tests/test_helpers.h @@ -67,6 +67,7 @@ extern int cmp_files(const char *a, const char *b); extern int copy_file(const char *source, const char *dest); extern int rmdir_recurs(const char *directory_path); extern int copydir_recurs(const char *source_directory_path, const char *destination_directory_path); +extern int remove_placeholders(char *directory_path, char *filename); extern int open_temp_repo(git_repository **repo, const char *path); extern void close_temp_repo(git_repository *repo); From 2ce44b67f3398ff3c1bfdfd60a99f72bd0c42e90 Mon Sep 17 00:00:00 2001 From: nulltoken Date: Sat, 5 Mar 2011 14:33:31 +0100 Subject: [PATCH 49/81] Add test exercising the opening of an empty bare repository initialized by git --- tests/t12-repo.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/tests/t12-repo.c b/tests/t12-repo.c index 38e350958..b81b598fb 100644 --- a/tests/t12-repo.c +++ b/tests/t12-repo.c @@ -184,6 +184,21 @@ BEGIN_TEST(init2, "Initialize and open a bare repo with a relative path escaping rmdir_recurs(TEMP_REPO_FOLDER); END_TEST +#define EMPTY_BARE_REPOSITORY_NAME "empty_bare.git" +#define EMPTY_BARE_REPOSITORY_FOLDER TEST_RESOURCES "/" EMPTY_BARE_REPOSITORY_NAME "/" + +BEGIN_TEST(open0, "Open a bare repository that has just been initialized by git") + git_repository *repo; + + must_pass(copydir_recurs(EMPTY_BARE_REPOSITORY_FOLDER, TEMP_REPO_FOLDER)); + must_pass(remove_placeholders(TEMP_REPO_FOLDER, "dummy-marker.txt")); + + must_pass(git_repository_open(&repo, TEMP_REPO_FOLDER)); + + git_repository_free(repo); + must_pass(rmdir_recurs(TEMP_REPO_FOLDER)); +END_TEST + BEGIN_TEST(open2, "Open a bare repository with a relative path escaping out of the current working directory") char new_current_workdir[GIT_PATH_MAX]; char current_workdir[GIT_PATH_MAX]; @@ -217,6 +232,7 @@ BEGIN_SUITE(repository) ADD_TEST(init0); ADD_TEST(init1); ADD_TEST(init2); + ADD_TEST(open0); ADD_TEST(open2); END_SUITE From a8375f532204f4c0c70a3f937b47000b61669725 Mon Sep 17 00:00:00 2001 From: nulltoken Date: Sat, 5 Mar 2011 15:09:01 +0100 Subject: [PATCH 50/81] Add test exercising the opening of an standard repository initialized by git --- tests/t12-repo.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/tests/t12-repo.c b/tests/t12-repo.c index b81b598fb..adf20cfd7 100644 --- a/tests/t12-repo.c +++ b/tests/t12-repo.c @@ -199,6 +199,24 @@ BEGIN_TEST(open0, "Open a bare repository that has just been initialized by git" must_pass(rmdir_recurs(TEMP_REPO_FOLDER)); END_TEST +#define SOURCE_EMPTY_REPOSITORY_NAME "empty_standard_repo/.gitted" +#define EMPTY_REPOSITORY_NAME "empty_standard_repo/.git" +#define EMPTY_REPOSITORY_FOLDER TEST_RESOURCES "/" SOURCE_EMPTY_REPOSITORY_NAME "/" +#define DEST_REPOSITORY_FOLDER TEMP_REPO_FOLDER DOT_GIT "/" + +BEGIN_TEST(open1, "Open a standard repository that has just been initialized by git") + git_repository *repo; + + must_pass(copydir_recurs(EMPTY_REPOSITORY_FOLDER, DEST_REPOSITORY_FOLDER)); + must_pass(remove_placeholders(DEST_REPOSITORY_FOLDER, "dummy-marker.txt")); + + must_pass(git_repository_open(&repo, DEST_REPOSITORY_FOLDER)); + + git_repository_free(repo); + must_pass(rmdir_recurs(TEMP_REPO_FOLDER)); +END_TEST + + BEGIN_TEST(open2, "Open a bare repository with a relative path escaping out of the current working directory") char new_current_workdir[GIT_PATH_MAX]; char current_workdir[GIT_PATH_MAX]; @@ -233,6 +251,7 @@ BEGIN_SUITE(repository) ADD_TEST(init1); ADD_TEST(init2); ADD_TEST(open0); + ADD_TEST(open1); ADD_TEST(open2); END_SUITE From 7c80c19e1dffb4421f91913bc79b9cb7596634a4 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 23 Mar 2011 01:58:18 +0200 Subject: [PATCH 51/81] Fix compilation in MinGW --- include/git2/types.h | 2 +- src/fileops.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/include/git2/types.h b/include/git2/types.h index db09f3842..88f6b7d55 100644 --- a/include/git2/types.h +++ b/include/git2/types.h @@ -57,7 +57,7 @@ typedef __time64_t git_time_t; #elif defined(__MINGW32__) typedef off64_t git_off_t; -typedef time64_t git_time_t; +typedef __time64_t git_time_t; #else /* POSIX */ diff --git a/src/fileops.c b/src/fileops.c index e440ea6ab..d754c49ee 100644 --- a/src/fileops.c +++ b/src/fileops.c @@ -32,7 +32,7 @@ static int creat_tempfile(char *path_out, const char *tmp_dir, const char *filen git__joinpath(path_out, tmp_dir, filename); strcat(path_out, "_git2_XXXXXX"); -#ifdef GIT_WIN32 +#if defined(_MSC_VER) /* FIXME: there may be race conditions when multi-threading * with the library */ if (_mktemp_s(path_out, GIT_PATH_MAX) != 0) From b0b8313583afdcd54a83ba3855cf3f6f287ae0ab Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Tue, 22 Mar 2011 16:15:50 +0100 Subject: [PATCH 52/81] Add close wappers for commit, tree, tag and blob MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit In the same spirit that git_repository_lookup is no longer available, add wrappers so the users don't have to cast when closing their objects. Signed-off-by: Carlos Martín Nieto --- include/git2/blob.h | 18 ++++++++++++++++++ include/git2/commit.h | 17 +++++++++++++++++ include/git2/tag.h | 18 ++++++++++++++++++ include/git2/tree.h | 18 ++++++++++++++++++ 4 files changed, 71 insertions(+) diff --git a/include/git2/blob.h b/include/git2/blob.h index 3cd1467bf..c38a52f65 100644 --- a/include/git2/blob.h +++ b/include/git2/blob.h @@ -52,6 +52,24 @@ GIT_INLINE(int) git_blob_lookup(git_blob **blob, git_repository *repo, const git return git_object_lookup((git_object **)blob, repo, id, GIT_OBJ_BLOB); } +/** + * Close an open blob + * + * This is a wrapper around git_object_close() + * + * IMPORTANT: + * It *is* necessary to call this method when you stop + * using a blob. Failure to do so will cause a memory leak. + * + * @param blob the blob to close + */ + +GIT_INLINE(void) git_blob_close(git_blob *blob) +{ + return git_object_close((git_object *) blob); +} + + /** * Get a read-only buffer with the raw content of a blob. * diff --git a/include/git2/commit.h b/include/git2/commit.h index 8306eb152..d497324c6 100644 --- a/include/git2/commit.h +++ b/include/git2/commit.h @@ -53,6 +53,23 @@ GIT_INLINE(int) git_commit_lookup(git_commit **commit, git_repository *repo, con return git_object_lookup((git_object **)commit, repo, id, GIT_OBJ_COMMIT); } +/** + * Close an open commit + * + * This is a wrapper around git_object_close() + * + * IMPORTANT: + * It *is* necessary to call this method when you stop + * using a commit. Failure to do so will cause a memory leak. + * + * @param commit the commit to close + */ + +GIT_INLINE(void) git_commit_close(git_commit *commit) +{ + return git_object_close((git_object *) commit); +} + /** * Get the id of a commit. * diff --git a/include/git2/tag.h b/include/git2/tag.h index c343f6bf4..845b20516 100644 --- a/include/git2/tag.h +++ b/include/git2/tag.h @@ -52,6 +52,24 @@ GIT_INLINE(int) git_tag_lookup(git_tag **tag, git_repository *repo, const git_oi return git_object_lookup((git_object **)tag, repo, id, (git_otype)GIT_OBJ_TAG); } +/** + * Close an open tag + * + * This is a wrapper around git_object_close() + * + * IMPORTANT: + * It *is* necessary to call this method when you stop + * using a tag. Failure to do so will cause a memory leak. + * + * @param tag the tag to close + */ + +GIT_INLINE(void) git_tag_close(git_tag *tag) +{ + return git_object_close((git_object *) tag); +} + + /** * Get the id of a tag. * diff --git a/include/git2/tree.h b/include/git2/tree.h index ec2b51646..cb959a790 100644 --- a/include/git2/tree.h +++ b/include/git2/tree.h @@ -52,6 +52,24 @@ GIT_INLINE(int) git_tree_lookup(git_tree **tree, git_repository *repo, const git return git_object_lookup((git_object **)tree, repo, id, GIT_OBJ_TREE); } +/** + * Close an open tree + * + * This is a wrapper around git_object_close() + * + * IMPORTANT: + * It *is* necessary to call this method when you stop + * using a tree. Failure to do so will cause a memory leak. + * + * @param tree the tree to close + */ + +GIT_INLINE(void) git_tree_close(git_tree *tree) +{ + return git_object_close((git_object *) tree); +} + + /** * Get the id of a tree. * From f0d08b7cec782634e550ad0cd52a705b2c87244c Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 23 Mar 2011 15:40:47 +0200 Subject: [PATCH 53/81] Remove `git_repository_gc` from the headers --- include/git2/repository.h | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/include/git2/repository.h b/include/git2/repository.h index 317b367d2..00c1f20d0 100644 --- a/include/git2/repository.h +++ b/include/git2/repository.h @@ -155,20 +155,16 @@ GIT_EXTERN(int) git_repository_index(git_index **index, git_repository *repo); /** * Free a previously allocated repository * + * Note that after a repository is free'd, all the objects it has spawned + * will still exist until they are manually closed by the user + * with `git_object_close`, but accessing any of the attributes of + * an object without a backing repository will result in undefined + * behavior + * * @param repo repository handle to close. If NULL nothing occurs. */ GIT_EXTERN(void) git_repository_free(git_repository *repo); -/** - * Force a garbage collector pass on the repository - * - * This will force-free any cached objects that have been - * previously marked by the user as closed (`git_object_close`). - * - * @param repo repository handle to collect. If NULL nothing occurs. - */ -GIT_EXTERN(int) git_repository_gc(git_repository *repo); - /** * Creates a new Git repository in the given folder. * From c0ffe518539073cac6b54249fa11d380ba85049c Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 23 Mar 2011 15:44:52 +0200 Subject: [PATCH 54/81] Do not return on `void` helper methods MSVC doesn't swallow that. --- include/git2/blob.h | 2 +- include/git2/commit.h | 2 +- include/git2/tag.h | 2 +- include/git2/tree.h | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/git2/blob.h b/include/git2/blob.h index c38a52f65..0e05d6f89 100644 --- a/include/git2/blob.h +++ b/include/git2/blob.h @@ -66,7 +66,7 @@ GIT_INLINE(int) git_blob_lookup(git_blob **blob, git_repository *repo, const git GIT_INLINE(void) git_blob_close(git_blob *blob) { - return git_object_close((git_object *) blob); + git_object_close((git_object *) blob); } diff --git a/include/git2/commit.h b/include/git2/commit.h index d497324c6..c09b34843 100644 --- a/include/git2/commit.h +++ b/include/git2/commit.h @@ -67,7 +67,7 @@ GIT_INLINE(int) git_commit_lookup(git_commit **commit, git_repository *repo, con GIT_INLINE(void) git_commit_close(git_commit *commit) { - return git_object_close((git_object *) commit); + git_object_close((git_object *) commit); } /** diff --git a/include/git2/tag.h b/include/git2/tag.h index 845b20516..ee92cd5c2 100644 --- a/include/git2/tag.h +++ b/include/git2/tag.h @@ -66,7 +66,7 @@ GIT_INLINE(int) git_tag_lookup(git_tag **tag, git_repository *repo, const git_oi GIT_INLINE(void) git_tag_close(git_tag *tag) { - return git_object_close((git_object *) tag); + git_object_close((git_object *) tag); } diff --git a/include/git2/tree.h b/include/git2/tree.h index cb959a790..164aec9e2 100644 --- a/include/git2/tree.h +++ b/include/git2/tree.h @@ -66,7 +66,7 @@ GIT_INLINE(int) git_tree_lookup(git_tree **tree, git_repository *repo, const git GIT_INLINE(void) git_tree_close(git_tree *tree) { - return git_object_close((git_object *) tree); + git_object_close((git_object *) tree); } From 08db1efd3d64bda358071612ff3662f4bf1aea61 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 23 Mar 2011 15:53:12 +0200 Subject: [PATCH 55/81] Add Ocaml bindings to the readme --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 05378685b..8f860eb22 100644 --- a/README.md +++ b/README.md @@ -128,6 +128,7 @@ Here are the bindings to libgit2 that are currently available: * node-gitteh (Node.js bindings) * nodegit (Node.js bindings) * libqgit2 (C++ QT bindings) +* libgit2-ocaml (ocaml bindings) * Geef (Erlang bindings) If you start another language binding to libgit2, please let us know so From f6f72d7ef8091bf1fcf19f284e1db62a43f93381 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 23 Mar 2011 18:44:53 +0200 Subject: [PATCH 56/81] Improve the ODB writing backend Temporary files when doing streaming writes are now stored inside the Objects folder, to prevent issues when moving files between disks/partitions. Add support for block writes to the ODB again (for those backends that cannot implement streaming). --- include/git2/odb.h | 20 ++++++++++++ include/git2/odb_backend.h | 7 +++++ src/filebuf.c | 11 ++----- src/fileops.c | 64 ++------------------------------------ src/fileops.h | 2 +- src/odb.c | 35 +++++++++++++++++++++ src/odb_loose.c | 6 ++-- 7 files changed, 71 insertions(+), 74 deletions(-) diff --git a/include/git2/odb.h b/include/git2/odb.h index 1a0f0e64c..aa48b053a 100644 --- a/include/git2/odb.h +++ b/include/git2/odb.h @@ -156,6 +156,26 @@ GIT_EXTERN(int) git_odb_read_header(size_t *len_p, git_otype *type_p, git_odb *d */ GIT_EXTERN(int) git_odb_exists(git_odb *db, const git_oid *id); +/** + * Write an object directly into the ODB + * + * This method writes a full object straight into the ODB. + * For most cases, it is preferred to write objects through a write + * stream, which is both faster and less memory intensive, specially + * for big objects. + * + * This method is provided for compatibility with custom backends + * which are not able to support streaming writes + * + * @param oid pointer to store the OID result of the write + * @param odb object database where to store the object + * @param data buffer with the data to storr + * @param len size of the buffer + * @param type type of the data to store + * @return 0 on success; error code otherwise + */ +GIT_EXTERN(int) git_odb_write(git_oid *oid, git_odb *odb, const void *data, size_t len, git_otype type); + /** * Open a stream to write an object into the ODB * diff --git a/include/git2/odb_backend.h b/include/git2/odb_backend.h index 3875ec7f6..7eadef33e 100644 --- a/include/git2/odb_backend.h +++ b/include/git2/odb_backend.h @@ -55,6 +55,13 @@ struct git_odb_backend { struct git_odb_backend *, const git_oid *); + int (* write)( + git_oid *, + struct git_odb_backend *, + const void *, + size_t, + git_otype); + int (* writestream)( struct git_odb_stream **, struct git_odb_backend *, diff --git a/src/filebuf.c b/src/filebuf.c index 607ad618d..dff9373f6 100644 --- a/src/filebuf.c +++ b/src/filebuf.c @@ -151,8 +151,7 @@ int git_filebuf_open(git_filebuf *file, const char *path, int flags) int error; size_t path_len; - if (file == NULL) - return GIT_ERROR; + assert(file && path); memset(file, 0x0, sizeof(git_filebuf)); @@ -203,7 +202,7 @@ int git_filebuf_open(git_filebuf *file, const char *path, int flags) char tmp_path[GIT_PATH_MAX]; /* Open the file as temporary for locking */ - file->fd = gitfo_creat_tmp(tmp_path, "_filebuf_"); + file->fd = gitfo_mktemp(tmp_path, path); if (file->fd < 0) { error = GIT_EOSERR; goto cleanup; @@ -218,12 +217,6 @@ int git_filebuf_open(git_filebuf *file, const char *path, int flags) goto cleanup; } } else { - /* If the file is not temporary, make sure we have a path */ - if (path == NULL) { - error = GIT_ERROR; - goto cleanup; - } - path_len = strlen(path); /* Save the original path of the file */ diff --git a/src/fileops.c b/src/fileops.c index d754c49ee..5dd4a3806 100644 --- a/src/fileops.c +++ b/src/fileops.c @@ -25,11 +25,11 @@ int gitfo_mkdir_2file(const char *file_path) return GIT_SUCCESS; } -static int creat_tempfile(char *path_out, const char *tmp_dir, const char *filename) +int gitfo_mktemp(char *path_out, const char *filename) { int fd; - git__joinpath(path_out, tmp_dir, filename); + strcpy(path_out, filename); strcat(path_out, "_git2_XXXXXX"); #if defined(_MSC_VER) @@ -46,66 +46,6 @@ static int creat_tempfile(char *path_out, const char *tmp_dir, const char *filen return fd >= 0 ? fd : GIT_EOSERR; } -static const char *find_tmpdir(void) -{ - static int tmpdir_not_found = 0; - static char temp_dir[GIT_PATH_MAX]; - static const char *env_vars[] = { - "TEMP", "TMP", "TMPDIR" - }; - - unsigned int i, j; - char test_file[GIT_PATH_MAX]; - - if (tmpdir_not_found) - return NULL; - - if (temp_dir[0] != '\0') - return temp_dir; - - for (i = 0; i < ARRAY_SIZE(env_vars); ++i) { - char *env_path; - - env_path = getenv(env_vars[i]); - if (env_path == NULL) - continue; - - strcpy(temp_dir, env_path); - - /* Fix backslashes because Windows environment vars - * are probably fucked up */ - for (j = 0; j < strlen(temp_dir); ++j) - if (temp_dir[j] == '\\') - temp_dir[j] = '/'; - - if (creat_tempfile(test_file, temp_dir, "writetest") >= 0) { - gitfo_unlink(test_file); - return temp_dir; - } - } - - /* last resort: current folder. */ - strcpy(temp_dir, "./"); - if (creat_tempfile(test_file, temp_dir, "writetest") >= 0) { - gitfo_unlink(test_file); - return temp_dir; - } - - tmpdir_not_found = 1; - return NULL; -} - -int gitfo_creat_tmp(char *path_out, const char *filename) -{ - const char *tmp_dir; - - tmp_dir = find_tmpdir(); - if (tmp_dir == NULL) - return GIT_EOSERR; - - return creat_tempfile(path_out, tmp_dir, filename); -} - int gitfo_open(const char *path, int flags) { int fd = open(path, flags | O_BINARY); diff --git a/src/fileops.h b/src/fileops.h index ce236f608..6e0fd9d14 100644 --- a/src/fileops.h +++ b/src/fileops.h @@ -58,7 +58,7 @@ extern int gitfo_exists(const char *path); extern int gitfo_open(const char *path, int flags); extern int gitfo_creat(const char *path, int mode); extern int gitfo_creat_force(const char *path, int mode); -extern int gitfo_creat_tmp(char *path_out, const char *filename); +extern int gitfo_mktemp(char *path_out, const char *filename); extern int gitfo_isdir(const char *path); extern int gitfo_mkdir_recurs(const char *path, int mode); extern int gitfo_mkdir_2file(const char *path); diff --git a/src/odb.c b/src/odb.c index d825fd95c..33d5468d9 100644 --- a/src/odb.c +++ b/src/odb.c @@ -415,6 +415,41 @@ int git_odb_read(git_odb_object **out, git_odb *db, const git_oid *id) return error; } +int git_odb_write(git_oid *oid, git_odb *db, const void *data, size_t len, git_otype type) +{ + unsigned int i; + int error = GIT_ERROR; + + assert(oid && db); + + for (i = 0; i < db->backends.length && error < 0; ++i) { + backend_internal *internal = git_vector_get(&db->backends, i); + git_odb_backend *b = internal->backend; + + /* we don't write in alternates! */ + if (internal->is_alternate) + continue; + + if (b->write != NULL) + error = b->write(oid, b, data, len, type); + } + + /* if no backends were able to write the object directly, we try a streaming + * write to the backends; just write the whole object into the stream in one + * push */ + if (error < GIT_SUCCESS) { + git_odb_stream *stream; + + if ((error = git_odb_open_wstream(&stream, db, len, type)) == GIT_SUCCESS) { + stream->write(stream, data, len); + error = stream->finalize_write(oid, stream); + stream->free(stream); + } + } + + return error; +} + int git_odb_open_wstream(git_odb_stream **stream, git_odb *db, size_t size, git_otype type) { unsigned int i; diff --git a/src/odb_loose.c b/src/odb_loose.c index 4ab1128f3..8ee01cd2c 100644 --- a/src/odb_loose.c +++ b/src/odb_loose.c @@ -579,7 +579,7 @@ int loose_backend__stream(git_odb_stream **stream_out, git_odb_backend *_backend loose_backend *backend; loose_writestream *stream; - char hdr[64]; + char hdr[64], tmp_path[GIT_PATH_MAX]; int hdrlen; int error; @@ -603,7 +603,9 @@ int loose_backend__stream(git_odb_stream **stream_out, git_odb_backend *_backend stream->stream.free = &loose_backend__stream_free; stream->stream.mode = GIT_STREAM_WRONLY; - error = git_filebuf_open(&stream->fbuf, NULL, + git__joinpath(tmp_path, backend->objects_dir, "tmp_object"); + + error = git_filebuf_open(&stream->fbuf, tmp_path, GIT_FILEBUF_HASH_CONTENTS | GIT_FILEBUF_DEFLATE_CONTENTS | GIT_FILEBUF_TEMPORARY); From c585f55ff454bfc2d97e2db6b10fd8eab31b119d Mon Sep 17 00:00:00 2001 From: Jakob Pfender Date: Wed, 23 Mar 2011 12:35:08 +0100 Subject: [PATCH 57/81] common.h: Fix minor typos Fix a few minor typos in the documentation of the GIT_ERROR codes. --- include/git2/common.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/include/git2/common.h b/include/git2/common.h index 7cfb8982e..57ab9c1ff 100644 --- a/include/git2/common.h +++ b/include/git2/common.h @@ -119,13 +119,13 @@ /** The object type is invalid or doesn't match */ #define GIT_EINVALIDTYPE (GIT_ERROR - 8) -/** The object cannot be written that because it's missing internal data */ +/** The object cannot be written because it's missing internal data */ #define GIT_EMISSINGOBJDATA (GIT_ERROR - 9) /** The packfile for the ODB is corrupted */ #define GIT_EPACKCORRUPTED (GIT_ERROR - 10) -/** Failed to adquire or release a file lock */ +/** Failed to acquire or release a file lock */ #define GIT_EFLOCKFAIL (GIT_ERROR - 11) /** The Z library failed to inflate/deflate an object's data */ @@ -146,7 +146,7 @@ /** The specified symbolic reference is too deeply nested */ #define GIT_ETOONESTEDSYMREF (GIT_ERROR - 17) -/** The pack-refs file is either corrupted of its format is not currently supported */ +/** The pack-refs file is either corrupted or its format is not currently supported */ #define GIT_EPACKEDREFSCORRUPTED (GIT_ERROR - 18) /** The path is invalid */ From ea269511adee9db7343b802832626dba3f9c382e Mon Sep 17 00:00:00 2001 From: Jakob Pfender Date: Wed, 23 Mar 2011 14:57:41 +0100 Subject: [PATCH 58/81] odb.h: Fix minor typo Fix a doxygen typo ("@para" instead of "@param") in odb.h --- include/git2/odb.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/git2/odb.h b/include/git2/odb.h index aa48b053a..1d351beea 100644 --- a/include/git2/odb.h +++ b/include/git2/odb.h @@ -200,7 +200,7 @@ GIT_EXTERN(int) git_odb_write(git_oid *oid, git_odb *odb, const void *data, size * @param stream pointer where to store the stream * @param db object database where the stream will write * @param size final size of the object that will be written - * @para type type of the object that will be written + * @param type type of the object that will be written * @return 0 if the stream was created; error code otherwise */ GIT_EXTERN(int) git_odb_open_wstream(git_odb_stream **stream, git_odb *db, size_t size, git_otype type); From 815c9bc7debb7493f18cb7bca1febc11f5ea3ea6 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Wed, 23 Mar 2011 20:18:34 +0200 Subject: [PATCH 59/81] Remove circular dependency in includes --- include/git2/odb_backend.h | 1 - 1 file changed, 1 deletion(-) diff --git a/include/git2/odb_backend.h b/include/git2/odb_backend.h index 7eadef33e..ba41f726c 100644 --- a/include/git2/odb_backend.h +++ b/include/git2/odb_backend.h @@ -28,7 +28,6 @@ #include "common.h" #include "types.h" #include "oid.h" -#include "odb.h" /** * @file git2/backend.h From f3cf7f6df1d2e2fcbd4f22dae318fbc4570833d8 Mon Sep 17 00:00:00 2001 From: Scott Chacon Date: Wed, 23 Mar 2011 17:06:49 -0700 Subject: [PATCH 60/81] add go-git (Go bindings) to the readme --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 8f860eb22..1254adcd9 100644 --- a/README.md +++ b/README.md @@ -127,6 +127,7 @@ Here are the bindings to libgit2 that are currently available: * GitForDelphi (Delphi bindings) * node-gitteh (Node.js bindings) * nodegit (Node.js bindings) +* go-git (Go bindings) * libqgit2 (C++ QT bindings) * libgit2-ocaml (ocaml bindings) * Geef (Erlang bindings) From 3bdc0d4c8c3a51e6b338c8166d98d5b82cb01a43 Mon Sep 17 00:00:00 2001 From: Jakob Pfender Date: Thu, 24 Mar 2011 15:32:24 +0100 Subject: [PATCH 61/81] index.c: Read index after initialization The current behaviour of git_index_open{bare,inrepo}() is unexpected. When an index is opened, an in-memory index object is created that is linked to the index discovered by git_repository_open(). However, this index object is empty, as the on-disk index is not read. To fully open the on-disk index file, git_index_read() has to be called. This leads to confusing behaviour. Consider the following code: git_index *idx; git_index_open_inrepo(&idx, repo); git_index_write(idx); You would expect this to have no effect, as the index is never ostensibly manipulated. However, what actually happens is that the index entries are removed from the on-disk index because the empty in-memory index object created by open_inrepo() is written back to the disk. This patch reads the index after opening it. --- src/index.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/index.c b/src/index.c index 34208731c..7e4d6bbc7 100644 --- a/src/index.c +++ b/src/index.c @@ -148,7 +148,7 @@ static int index_initialize(git_index **index_out, git_repository *owner, const index->on_disk = 1; *index_out = index; - return GIT_SUCCESS; + return git_index_read(index); } int git_index_open_bare(git_index **index_out, const char *index_path) From 12f6d8e1453eacc1b4c6db0f6071541bcdf2e0e0 Mon Sep 17 00:00:00 2001 From: Jakob Pfender Date: Thu, 24 Mar 2011 13:48:22 +0100 Subject: [PATCH 62/81] index.h: Correct documentation for git_index_open_inrepo() Fix the doxygen comments for git_index_open_inrepo(). Previously they referred to a param index_path and omitted index (probably a c&p error). --- include/git2/index.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/git2/index.h b/include/git2/index.h index 605740c10..7bb0c9529 100644 --- a/include/git2/index.h +++ b/include/git2/index.h @@ -91,8 +91,8 @@ GIT_EXTERN(int) git_index_open_bare(git_index **index, const char *index_path); * Open the Index inside the git repository pointed * by 'repo'. * + * @param index the pointer for the new index * @param repo the git repo which owns the index - * @param index_path the path to the index file in disk * @return 0 on success; error code otherwise */ GIT_EXTERN(int) git_index_open_inrepo(git_index **index, git_repository *repo); From 051d6915d70e6fec597fe1bfdee2870561f523d1 Mon Sep 17 00:00:00 2001 From: Jakob Pfender Date: Thu, 24 Mar 2011 18:28:04 +0100 Subject: [PATCH 63/81] index.h: Fix minor typo --- include/git2/index.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/git2/index.h b/include/git2/index.h index 7bb0c9529..599512f8a 100644 --- a/include/git2/index.h +++ b/include/git2/index.h @@ -132,7 +132,7 @@ GIT_EXTERN(int) git_index_read(git_index *index); GIT_EXTERN(int) git_index_write(git_index *index); /** - * Find the first index of any entires which point to given + * Find the first index of any entries which point to given * path in the Git index. * * @param index an existing index object From 09e8de0f38ed8f28c02e5a3fa987cbb74e98a671 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Fri, 25 Mar 2011 23:53:38 +0200 Subject: [PATCH 64/81] New external API method: `git_reference_listcb` List all the references in the repository, calling a custom callback for each one. The listed references may be filtered by type, or using a bitwise OR of several types. Use the magic value `GIT_REF_LISTALL` to obtain all references, including packed ones. The `callback` function will be called for each of the references in the repository, and will receive the name of the reference and the `payload` value passed to this method. --- include/git2/refs.h | 23 ++++++++++++++++ src/refs.c | 66 ++++++++++++++++++++++++++++++--------------- 2 files changed, 68 insertions(+), 21 deletions(-) diff --git a/include/git2/refs.h b/include/git2/refs.h index 4ffc5ce5b..da55eaa3b 100644 --- a/include/git2/refs.h +++ b/include/git2/refs.h @@ -241,6 +241,29 @@ GIT_EXTERN(int) git_reference_packall(git_repository *repo); */ GIT_EXTERN(int) git_reference_listall(git_strarray *array, git_repository *repo, unsigned int list_flags); + +/** + * List all the references in the repository, calling a custom + * callback for each one. + * + * The listed references may be filtered by type, or using + * a bitwise OR of several types. Use the magic value + * `GIT_REF_LISTALL` to obtain all references, including + * packed ones. + * + * The `callback` function will be called for each of the references + * in the repository, and will receive the name of the reference and + * the `payload` value passed to this method. + * + * @param repo Repository where to find the refs + * @param list_flags Filtering flags for the reference + * listing. + * @param callback Function which will be called for every listed ref + * @param payload Additional data to pass to the callback + * @return 0 on success; error code otherwise + */ +GIT_EXTERN(int) git_reference_listcb(git_repository *repo, unsigned int list_flags, int (*callback)(const char *, void *), void *payload); + /** @} */ GIT_END_DECL #endif diff --git a/src/refs.c b/src/refs.c index 8e2965528..16bd74149 100644 --- a/src/refs.c +++ b/src/refs.c @@ -606,10 +606,12 @@ cleanup: struct dirent_list_data { - git_vector ref_list; git_repository *repo; size_t repo_path_len; unsigned int list_flags; + + int (*callback)(const char *, void *); + void *callback_payload; }; static int _dirent_loose_listall(void *_data, char *full_path) @@ -625,10 +627,12 @@ static int _dirent_loose_listall(void *_data, char *full_path) git_hashtable_lookup(data->repo->references.packfile, file_path) != NULL) return GIT_SUCCESS; - if ((data->list_flags & loose_guess_rtype(full_path)) == 0) - return GIT_SUCCESS; /* we are filtering out this reference */ + if (data->list_flags != GIT_REF_LISTALL) { + if ((data->list_flags & loose_guess_rtype(full_path)) == 0) + return GIT_SUCCESS; /* we are filtering out this reference */ + } - return git_vector_insert(&data->ref_list, git__strdup(file_path)); + return data->callback(file_path, data->callback_payload); } static int _dirent_loose_load(void *data, char *full_path) @@ -1402,47 +1406,67 @@ int git_reference_packall(git_repository *repo) return packed_write(repo); } -int git_reference_listall(git_strarray *array, git_repository *repo, unsigned int list_flags) +int git_reference_listcb(git_repository *repo, unsigned int list_flags, int (*callback)(const char *, void *), void *payload) { int error; struct dirent_list_data data; char refs_path[GIT_PATH_MAX]; - array->strings = NULL; - array->count = 0; - - git_vector_init(&data.ref_list, 8, NULL); - data.repo_path_len = strlen(repo->path_repository); - data.list_flags = list_flags; - data.repo = repo; - /* list all the packed references first */ if (list_flags & GIT_REF_PACKED) { const char *ref_name; void *_unused; - if ((error = packed_load(repo)) < GIT_SUCCESS) { - git_vector_free(&data.ref_list); + if ((error = packed_load(repo)) < GIT_SUCCESS) return error; - } GIT_HASHTABLE_FOREACH(repo->references.packfile, ref_name, _unused, - git_vector_insert(&data.ref_list, git__strdup(ref_name)); + if ((error = callback(ref_name, payload)) < GIT_SUCCESS) + return error; ); } /* now list the loose references, trying not to * duplicate the ref names already in the packed-refs file */ + + data.repo_path_len = strlen(repo->path_repository); + data.list_flags = list_flags; + data.repo = repo; + data.callback = callback; + data.callback_payload = payload; + + git__joinpath(refs_path, repo->path_repository, GIT_REFS_DIR); - error = gitfo_dirent(refs_path, GIT_PATH_MAX, _dirent_loose_listall, &data); + return gitfo_dirent(refs_path, GIT_PATH_MAX, _dirent_loose_listall, &data); +} + +int cb__reflist_add(const char *ref, void *data) +{ + return git_vector_insert((git_vector *)data, git__strdup(ref)); +} + +int git_reference_listall(git_strarray *array, git_repository *repo, unsigned int list_flags) +{ + int error; + git_vector ref_list; + + assert(array && repo); + + array->strings = NULL; + array->count = 0; + + if (git_vector_init(&ref_list, 8, NULL) < GIT_SUCCESS) + return GIT_ENOMEM; + + error = git_reference_listcb(repo, list_flags, &cb__reflist_add, (void *)&ref_list); if (error < GIT_SUCCESS) { - git_vector_free(&data.ref_list); + git_vector_free(&ref_list); return error; } - array->strings = (char **)data.ref_list.contents; - array->count = data.ref_list.length; + array->strings = (char **)ref_list.contents; + array->count = ref_list.length; return GIT_SUCCESS; } From 9d80d74da58207da65eebc4f5c1919fa382d8a38 Mon Sep 17 00:00:00 2001 From: schu Date: Mon, 28 Mar 2011 17:57:08 +0200 Subject: [PATCH 65/81] signature.h: Fix tiny typo --- include/git2/signature.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/git2/signature.h b/include/git2/signature.h index 200397ab3..40412a45f 100644 --- a/include/git2/signature.h +++ b/include/git2/signature.h @@ -45,7 +45,7 @@ GIT_BEGIN_DECL * @email email of the person * @time time when the action happened * @offset timezone offset in minutes for the time - * @return the new sig, NULl on out of memory + * @return the new sig, NULL on out of memory */ GIT_EXTERN(git_signature *) git_signature_new(const char *name, const char *email, git_time_t time, int offset); From 683581a3e91032d9fdc203933d691b9353f24492 Mon Sep 17 00:00:00 2001 From: schu Date: Mon, 28 Mar 2011 17:59:13 +0200 Subject: [PATCH 66/81] index.c: Fix tiny typos --- src/index.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/index.c b/src/index.c index 7e4d6bbc7..6a31dd5cb 100644 --- a/src/index.c +++ b/src/index.c @@ -74,7 +74,7 @@ struct entry_short { uint32_t file_size; git_oid oid; uint16_t flags; - char path[1]; /* arbritrary length */ + char path[1]; /* arbitrary length */ }; struct entry_long { @@ -89,7 +89,7 @@ struct entry_long { git_oid oid; uint16_t flags; uint16_t flags_extended; - char path[1]; /* arbritrary length */ + char path[1]; /* arbitrary length */ }; /* local declarations */ From 483526ebec8b50682b6edd3e1db92df7569ca7b0 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Mon, 28 Mar 2011 22:23:44 +0300 Subject: [PATCH 67/81] Update the SQLite backend --- src/backends/sqlite.c | 38 +++++++++++++++++--------------------- tests/t11-sqlite.c | 21 ++++++++++++--------- 2 files changed, 29 insertions(+), 30 deletions(-) diff --git a/src/backends/sqlite.c b/src/backends/sqlite.c index 72d7b4d8e..a4c6d4825 100644 --- a/src/backends/sqlite.c +++ b/src/backends/sqlite.c @@ -44,21 +44,20 @@ typedef struct { sqlite3_stmt *st_read_header; } sqlite_backend; -int sqlite_backend__read_header(git_rawobj *obj, git_odb_backend *_backend, const git_oid *oid) +int sqlite_backend__read_header(size_t *len_p, git_otype *type_p, git_odb_backend *_backend, const git_oid *oid) { sqlite_backend *backend; int error; - assert(obj && _backend && oid); + assert(len_p && type_p && _backend && oid); backend = (sqlite_backend *)_backend; error = GIT_ERROR; - obj->data = NULL; if (sqlite3_bind_text(backend->st_read_header, 1, (char *)oid->id, 20, SQLITE_TRANSIENT) == SQLITE_OK) { if (sqlite3_step(backend->st_read_header) == SQLITE_ROW) { - obj->type = sqlite3_column_int(backend->st_read_header, 0); - obj->len = sqlite3_column_int(backend->st_read_header, 1); + *type_p = (git_otype)sqlite3_column_int(backend->st_read_header, 0); + *len_p = (size_t)sqlite3_column_int(backend->st_read_header, 1); assert(sqlite3_step(backend->st_read_header) == SQLITE_DONE); error = GIT_SUCCESS; } else { @@ -71,26 +70,26 @@ int sqlite_backend__read_header(git_rawobj *obj, git_odb_backend *_backend, cons } -int sqlite_backend__read(git_rawobj *obj, git_odb_backend *_backend, const git_oid *oid) +int sqlite_backend__read(void **data_p, size_t *len_p, git_otype *type_p, git_odb_backend *_backend, const git_oid *oid) { sqlite_backend *backend; int error; - assert(obj && _backend && oid); + assert(data_p && len_p && type_p && _backend && oid); backend = (sqlite_backend *)_backend; error = GIT_ERROR; if (sqlite3_bind_text(backend->st_read, 1, (char *)oid->id, 20, SQLITE_TRANSIENT) == SQLITE_OK) { if (sqlite3_step(backend->st_read) == SQLITE_ROW) { - obj->type = sqlite3_column_int(backend->st_read, 0); - obj->len = sqlite3_column_int(backend->st_read, 1); - obj->data = git__malloc(obj->len); + *type_p = (git_otype)sqlite3_column_int(backend->st_read, 0); + *len_p = (size_t)sqlite3_column_int(backend->st_read, 1); + *data_p = git__malloc(*len_p); - if (obj->data == NULL) { + if (*data_p == NULL) { error = GIT_ENOMEM; } else { - memcpy(obj->data, sqlite3_column_blob(backend->st_read, 2), obj->len); + memcpy(*data_p, sqlite3_column_blob(backend->st_read, 2), *len_p); error = GIT_SUCCESS; } @@ -126,27 +125,24 @@ int sqlite_backend__exists(git_odb_backend *_backend, const git_oid *oid) } -int sqlite_backend__write(git_oid *id, git_odb_backend *_backend, git_rawobj *obj) +int sqlite_backend__write(git_oid *id, git_odb_backend *_backend, const void *data, size_t len, git_otype type) { - char hdr[64]; - int hdrlen; - int error; sqlite_backend *backend; - assert(id && _backend && obj); + assert(id && _backend && data); backend = (sqlite_backend *)_backend; - if ((error = git_odb__hash_obj(id, hdr, sizeof(hdr), &hdrlen, obj)) < 0) + if ((error = git_odb_hash(id, data, len, type)) < 0) return error; error = SQLITE_ERROR; if (sqlite3_bind_text(backend->st_write, 1, (char *)id->id, 20, SQLITE_TRANSIENT) == SQLITE_OK && - sqlite3_bind_int(backend->st_write, 2, (int)obj->type) == SQLITE_OK && - sqlite3_bind_int(backend->st_write, 3, obj->len) == SQLITE_OK && - sqlite3_bind_blob(backend->st_write, 4, obj->data, obj->len, SQLITE_TRANSIENT) == SQLITE_OK) { + sqlite3_bind_int(backend->st_write, 2, (int)type) == SQLITE_OK && + sqlite3_bind_int(backend->st_write, 3, len) == SQLITE_OK && + sqlite3_bind_blob(backend->st_write, 4, data, len, SQLITE_TRANSIENT) == SQLITE_OK) { error = sqlite3_step(backend->st_write); } diff --git a/tests/t11-sqlite.c b/tests/t11-sqlite.c index fecf7886f..61ecf98ac 100644 --- a/tests/t11-sqlite.c +++ b/tests/t11-sqlite.c @@ -23,7 +23,7 @@ * Boston, MA 02110-1301, USA. */ #include "test_lib.h" - +#include "odb.h" #ifdef GIT2_SQLITE_BACKEND #include "t03-data.h" @@ -31,14 +31,17 @@ #include "git2/odb_backend.h" -static int cmp_objects(raw_object *o1, raw_object *o2) +static int cmp_objects(git_odb_object *odb_obj, git_rawobj *raw) { - if (o1->type != o2->type) + if (raw->type != git_odb_object_type(odb_obj)) return -1; - if (o1->len != o2->len) + + if (raw->len != git_odb_object_size(odb_obj)) return -1; - if ((o1->len > 0) && (memcmp(o1->data, o2->data, o1->len) != 0)) + + if ((raw->len > 0) && (memcmp(raw->data, git_odb_object_data(odb_obj), raw->len) != 0)) return -1; + return 0; } @@ -62,15 +65,15 @@ static git_odb *open_sqlite_odb(void) #define TEST_WRITE(PTR) {\ git_odb *db; \ git_oid id1, id2; \ - raw_object obj; \ + git_odb_object *obj; \ db = open_sqlite_odb(); \ must_be_true(db != NULL); \ must_pass(git_oid_mkstr(&id1, PTR.id)); \ - must_pass(git_odb_write(&id2, db, &PTR##_obj)); \ + must_pass(git_odb_write(&id2, db, PTR##_obj.data, PTR##_obj.len, PTR##_obj.type)); \ must_be_true(git_oid_cmp(&id1, &id2) == 0); \ must_pass(git_odb_read(&obj, db, &id1)); \ - must_pass(cmp_objects(&obj, &PTR##_obj)); \ - git_rawobj_close(&obj); \ + must_pass(cmp_objects(obj, &PTR##_obj)); \ + git_odb_object_close(obj); \ git_odb_close(db); \ } From 6dcb09b5b57875f334f61aebed695e2e4193db5e Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Mon, 28 Mar 2011 22:35:27 +0300 Subject: [PATCH 68/81] libgit2 version 0.11.0, "McSwifty" Apologies for the massive changes in the external API (that's my fault), and for the terrible codename for this release (that's @tclem's fault). The detailed overview for the major API changes can be found in the commit at 72a3fe42fb7208712bbe8f0981f4c6274c05e9c3. Major new features in this release: - Real caching and refcounting on parsed objects - Real caching and refcounting on objects read from the ODB - Streaming writes & reads from the ODB - Single-method writes for all object types - The external API is now partially thread-safe - Improved reference handling - New method to list references - ZLib is now built-in - Improvements to the Revision Walker - Tons of bug fixes Thanks to all the contributors who make this possible. Signed-off-by: Vicent Marti --- include/git2.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/git2.h b/include/git2.h index 248c2ba39..29fa98e18 100644 --- a/include/git2.h +++ b/include/git2.h @@ -26,7 +26,7 @@ #ifndef INCLUDE_git_git_h__ #define INCLUDE_git_git_h__ -#define LIBGIT2_VERSION "0.10.0" +#define LIBGIT2_VERSION "0.11.0" #define LIBGIT2_VER_MAJOR 0 #define LIBGIT2_VER_MINOR 10 #define LIBGIT2_VER_REVISION 0 From baad182cbcafb2f7332ad5864695a42a0354f684 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Mon, 28 Mar 2011 11:31:58 +0200 Subject: [PATCH 69/81] Add GIT_EEXISTS error code MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Carlos Martín Nieto --- include/git2/common.h | 3 +++ src/errors.c | 3 ++- 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/include/git2/common.h b/include/git2/common.h index 57ab9c1ff..7d3518f82 100644 --- a/include/git2/common.h +++ b/include/git2/common.h @@ -161,6 +161,9 @@ /** This feature has not been implemented yet */ #define GIT_ENOTIMPLEMENTED (GIT_ERROR - 22) +/** A refernce or object of the same name already exists */ +#define GIT_EEXISTS (GIT_ERROR - 23) + GIT_BEGIN_DECL typedef struct { diff --git a/src/errors.c b/src/errors.c index f6b964837..3c0e8eb1f 100644 --- a/src/errors.c +++ b/src/errors.c @@ -28,7 +28,8 @@ static struct { {GIT_EINVALIDPATH, "The path is invalid" }, {GIT_EREVWALKOVER, "The revision walker is empty; there are no more commits left to iterate"}, {GIT_EINVALIDREFSTATE, "The state of the reference is not valid"}, - {GIT_ENOTIMPLEMENTED, "This feature has not been implemented yet"} + {GIT_ENOTIMPLEMENTED, "This feature has not been implemented yet"}, + {GIT_EEXISTS, "A reference with this name already exists"} }; const char *git_strerror(int num) From fa204962006541bd9c111bd7f3fa37ed2f8a2d7e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Mon, 28 Mar 2011 12:00:50 +0200 Subject: [PATCH 70/81] Allow forcing the creation or renaming of references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add internal reference create and rename functions which take a force parameter, telling them to overwrite an existing reference if it exists. These functions try to update the reference if it's of the same type as the one it's going to be replaced by. Otherwise the old reference becomes invalid. Signed-off-by: Carlos Martín Nieto --- include/git2/refs.h | 56 ++++++ src/refs.c | 412 ++++++++++++++++++++++++++------------------ 2 files changed, 301 insertions(+), 167 deletions(-) diff --git a/include/git2/refs.h b/include/git2/refs.h index da55eaa3b..9ecf77295 100644 --- a/include/git2/refs.h +++ b/include/git2/refs.h @@ -68,6 +68,27 @@ GIT_EXTERN(int) git_reference_lookup(git_reference **reference_out, git_reposito */ GIT_EXTERN(int) git_reference_create_symbolic(git_reference **ref_out, git_repository *repo, const char *name, const char *target); +/** + * Create a new symbolic reference, overwriting an existing one with + * the same name, if it exists. + * + * If the new reference isn't a symbolic one, any pointers to the old + * reference become invalid. + * + * The reference will be created in the repository and written + * to the disk. + * + * This reference is owned by the repository and shall not + * be free'd by the user. + * + * @param ref_out Pointer to the newly created reference + * @param repo Repository where that reference will live + * @param name The name of the reference + * @param target The target of the reference + * @return 0 on success; error code otherwise + */ +GIT_EXTERN(int) git_reference_create_symbolic_force(git_reference **ref_out, git_repository *repo, const char *name, const char *target); + /** * Create a new object id reference. * @@ -85,6 +106,27 @@ GIT_EXTERN(int) git_reference_create_symbolic(git_reference **ref_out, git_repos */ GIT_EXTERN(int) git_reference_create_oid(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id); +/** + * Create a new object id reference, overwriting an existing one with + * the same name, if it exists. + * + * If the new reference isn't a symbolic one, any pointers to the old + * reference become invalid. + * + * The reference will be created in the repository and written + * to the disk. + * + * This reference is owned by the repository and shall not + * be free'd by the user. + * + * @param ref_out Pointer to the newly created reference + * @param repo Repository where that reference will live + * @param name The name of the reference + * @param id The object id pointed to by the reference. + * @return 0 on success; error code otherwise + */ +GIT_EXTERN(int) git_reference_create_oid_force(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id); + /** * Get the OID pointed to by a reference. * @@ -189,6 +231,20 @@ GIT_EXTERN(int) git_reference_set_oid(git_reference *ref, const git_oid *id); */ GIT_EXTERN(int) git_reference_rename(git_reference *ref, const char *new_name); +/** + * Rename an existing reference, overwriting an existing one with the + * same name, if it exists. + * + * This method works for both direct and symbolic references. + * The new name will be checked for validity and may be + * modified into a normalized form. + * + * The refernece will be immediately renamed in-memory + * and on disk. + * + */ +GIT_EXTERN(int) git_reference_rename_force(git_reference *ref, const char *new_name); + /** * Delete an existing reference * diff --git a/src/refs.c b/src/refs.c index 16bd74149..20c1d001f 100644 --- a/src/refs.c +++ b/src/refs.c @@ -915,8 +915,238 @@ cleanup: return error; } +/***************************************** + * Internal methods - reference creation + *****************************************/ + +int git_reference_create_symbolic_internal(git_reference **ref_out, git_repository *repo, const char *name, const char *target, int force) +{ + char normalized[MAX_GITDIR_TREE_STRUCTURE_PATH_LENGTH]; + int error = GIT_SUCCESS, updated = 0; + git_reference *ref = NULL, *old_ref = NULL; + + if (git_reference_lookup(&ref, repo, name) == GIT_SUCCESS && !force) + return GIT_EEXISTS; + + /* + * If they old ref was of the same type, then we can just update + * it (once we've checked that the target is valid). Otherwise we + * need a new reference because we can't make a symbolic ref out + * of an oid one. + * If if didn't exist, then we need to create a new one anyway. + */ + if (ref && ref->type & GIT_REF_SYMBOLIC){ + updated = 1; + } else { + ref = NULL; + error = reference_create(&ref, repo, name, GIT_REF_SYMBOLIC); + if (error < GIT_SUCCESS) + goto cleanup; + } + + /* The target can aither be the name of an object id reference or the name of another symbolic reference */ + error = normalize_name(normalized, target, 0); + if (error < GIT_SUCCESS) + goto cleanup; + + /* set the target; this will write the reference on disk */ + error = git_reference_set_target(ref, normalized); + if (error < GIT_SUCCESS) + goto cleanup; + + /* + * If we didn't update the ref, then we need to insert or replace + * it in the loose cache. If we replaced a ref, free it. + */ + if (!updated){ + error = git_hashtable_insert2(repo->references.loose_cache, ref->name, ref, (void **) &old_ref); + if (error < GIT_SUCCESS) + goto cleanup; + + if(old_ref) + reference_free(old_ref); + } + + *ref_out = ref; + + return error; + +cleanup: + reference_free(ref); + return error; +} + +int git_reference_create_oid_internal(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id, int force) +{ + int error = GIT_SUCCESS, updated = 0; + git_reference *ref = NULL, *old_ref = NULL; + + if(git_reference_lookup(&ref, repo, name) == GIT_SUCCESS && !force) + return GIT_EEXISTS; + + /* + * If they old ref was of the same type, then we can just update + * it (once we've checked that the target is valid). Otherwise we + * need a new reference because we can't make a symbolic ref out + * of an oid one. + * If if didn't exist, then we need to create a new one anyway. + */ + if (ref && ref-> type & GIT_REF_OID){ + updated = 1; + } else { + ref = NULL; + error = reference_create(&ref, repo, name, GIT_REF_OID); + if (error < GIT_SUCCESS) + goto cleanup; + } + + /* set the oid; this will write the reference on disk */ + error = git_reference_set_oid(ref, id); + if (error < GIT_SUCCESS) + goto cleanup; + + if(!updated){ + error = git_hashtable_insert2(repo->references.loose_cache, ref->name, ref, (void **) &old_ref); + if (error < GIT_SUCCESS) + goto cleanup; + + if(old_ref) + reference_free(old_ref); + } + + *ref_out = ref; + + return error; + +cleanup: + reference_free(ref); + return error; +} + +/* + * Rename a reference + * + * If the reference is packed, we need to rewrite the + * packfile to remove the reference from it and create + * the reference back as a loose one. + * + * If the reference is loose, we just rename it on + * the filesystem. + * + * We also need to re-insert the reference on its corresponding + * in-memory cache, since the caches are indexed by refname. + */ +int git_reference_rename_internal(git_reference *ref, const char *new_name, int force) +{ + int error; + char *old_name; + char old_path[GIT_PATH_MAX], new_path[GIT_PATH_MAX], normalized_name[MAX_GITDIR_TREE_STRUCTURE_PATH_LENGTH]; + git_reference *looked_up_ref, *old_ref = NULL; + + assert(ref); + + /* Ensure the name is valid */ + error = normalize_name(normalized_name, new_name, ref->type & GIT_REF_OID); + if (error < GIT_SUCCESS) + return error; + + /* Ensure we're not going to overwrite an existing reference */ + error = git_reference_lookup(&looked_up_ref, ref->owner, new_name); + if (error == GIT_SUCCESS && !force) + return GIT_EEXISTS; + + if (error != GIT_ENOTFOUND) + return error; + old_name = ref->name; + ref->name = git__strdup(new_name); + + if (ref->name == NULL) { + ref->name = old_name; + return GIT_ENOMEM; + } + + if (ref->type & GIT_REF_PACKED) { + /* write the packfile to disk; note + * that the state of the in-memory cache is not + * consistent, because the reference is indexed + * by its old name but it already has the new one. + * This doesn't affect writing, though, and allows + * us to rollback if writing fails + */ + + ref->type &= ~GIT_REF_PACKED; + + /* Create the loose ref under its new name */ + error = loose_write(ref); + if (error < GIT_SUCCESS) { + ref->type |= GIT_REF_PACKED; + goto cleanup; + } + + /* Remove from the packfile cache in order to avoid packing it back + * Note : we do not rely on git_reference_delete() because this would + * invalidate the reference. + */ + git_hashtable_remove(ref->owner->references.packfile, old_name); + + /* Recreate the packed-refs file without the reference */ + error = packed_write(ref->owner); + if (error < GIT_SUCCESS) + goto rename_loose_to_old_name; + + } else { + git__joinpath(old_path, ref->owner->path_repository, old_name); + git__joinpath(new_path, ref->owner->path_repository, ref->name); + + error = gitfo_mv_force(old_path, new_path); + if (error < GIT_SUCCESS) + goto cleanup; + + /* Once succesfully renamed, remove from the cache the reference known by its old name*/ + git_hashtable_remove(ref->owner->references.loose_cache, old_name); + } + + /* Store the renamed reference into the loose ref cache */ + error = git_hashtable_insert2(ref->owner->references.loose_cache, ref->name, ref, (void **) &old_ref); + + /* If we force-replaced, we need to free the old reference */ + if(old_ref) + reference_free(old_ref); + + free(old_name); + return error; + +cleanup: + /* restore the old name if this failed */ + free(ref->name); + ref->name = old_name; + return error; + +rename_loose_to_old_name: + /* If we hit this point. Something *bad* happened! Think "Ghostbusters + * crossing the streams" definition of bad. + * Either the packed-refs has been correctly generated and something else + * has gone wrong, or the writing of the new packed-refs has failed, and + * we're stuck with the old one. As a loose ref always takes priority over + * a packed ref, we'll eventually try and rename the generated loose ref to + * its former name. It even that fails, well... we might have lost the reference + * for good. :-/ + */ + + git__joinpath(old_path, ref->owner->path_repository, ref->name); + git__joinpath(new_path, ref->owner->path_repository, old_name); + + /* No error checking. We'll return the initial error */ + gitfo_mv_force(old_path, new_path); + + /* restore the old name */ + free(ref->name); + ref->name = old_name; + + return error; +} /***************************************** * External Library API @@ -975,64 +1205,23 @@ int git_reference_lookup(git_reference **ref_out, git_repository *repo, const ch int git_reference_create_symbolic(git_reference **ref_out, git_repository *repo, const char *name, const char *target) { - char normalized[MAX_GITDIR_TREE_STRUCTURE_PATH_LENGTH]; - int error = GIT_SUCCESS; - git_reference *ref = NULL; + return git_reference_create_symbolic_internal(ref_out, repo, name, target, 0); +} - error = reference_create(&ref, repo, name, GIT_REF_SYMBOLIC); - if (error < GIT_SUCCESS) - goto cleanup; - - /* The target can aither be the name of an object id reference or the name of another symbolic reference */ - error = normalize_name(normalized, target, 0); - if (error < GIT_SUCCESS) - goto cleanup; - - /* set the target; this will write the reference on disk */ - error = git_reference_set_target(ref, normalized); - if (error < GIT_SUCCESS) - goto cleanup; - - error = git_hashtable_insert(repo->references.loose_cache, ref->name, ref); - if (error < GIT_SUCCESS) - goto cleanup; - - *ref_out = ref; - - return error; - -cleanup: - reference_free(ref); - return error; +int git_reference_create_symbolic_force(git_reference **ref_out, git_repository *repo, const char *name, const char *target) +{ + return git_reference_create_symbolic_internal(ref_out, repo, name, target, 1); } int git_reference_create_oid(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id) { - int error = GIT_SUCCESS; - git_reference *ref = NULL; - - error = reference_create(&ref, repo, name, GIT_REF_OID); - if (error < GIT_SUCCESS) - goto cleanup; - - /* set the oid; this will write the reference on disk */ - error = git_reference_set_oid(ref, id); - if (error < GIT_SUCCESS) - goto cleanup; - - error = git_hashtable_insert(repo->references.loose_cache, ref->name, ref); - if (error < GIT_SUCCESS) - goto cleanup; - - *ref_out = ref; - - return error; - -cleanup: - reference_free(ref); - return error; + return git_reference_create_oid_internal(ref_out, repo, name, id, 0); } +int git_reference_create_oid_force(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id) +{ + return git_reference_create_oid_internal(ref_out, repo, name, id, 1); +} /** * Getters @@ -1240,125 +1429,14 @@ cleanup: return error; } -/* - * Rename a reference - * - * If the reference is packed, we need to rewrite the - * packfile to remove the reference from it and create - * the reference back as a loose one. - * - * If the reference is loose, we just rename it on - * the filesystem. - * - * We also need to re-insert the reference on its corresponding - * in-memory cache, since the caches are indexed by refname. - */ int git_reference_rename(git_reference *ref, const char *new_name) { - int error; - char *old_name; - char old_path[GIT_PATH_MAX], new_path[GIT_PATH_MAX], normalized_name[MAX_GITDIR_TREE_STRUCTURE_PATH_LENGTH]; - git_reference *looked_up_ref; + return git_reference_rename_internal(ref, new_name, 0); +} - assert(ref); - - /* Ensure the name is valid */ - error = normalize_name(normalized_name, new_name, ref->type & GIT_REF_OID); - if (error < GIT_SUCCESS) - return error; - - /* Ensure we're not going to overwrite an existing reference */ - error = git_reference_lookup(&looked_up_ref, ref->owner, new_name); - if (error == GIT_SUCCESS) - return GIT_EINVALIDREFNAME; - - if (error != GIT_ENOTFOUND) - return error; - - - old_name = ref->name; - ref->name = git__strdup(new_name); - - if (ref->name == NULL) { - ref->name = old_name; - return GIT_ENOMEM; - } - - if (ref->type & GIT_REF_PACKED) { - /* write the packfile to disk; note - * that the state of the in-memory cache is not - * consistent, because the reference is indexed - * by its old name but it already has the new one. - * This doesn't affect writing, though, and allows - * us to rollback if writing fails - */ - - ref->type &= ~GIT_REF_PACKED; - - /* Create the loose ref under its new name */ - error = loose_write(ref); - if (error < GIT_SUCCESS) { - ref->type |= GIT_REF_PACKED; - goto cleanup; - } - - /* Remove from the packfile cache in order to avoid packing it back - * Note : we do not rely on git_reference_delete() because this would - * invalidate the reference. - */ - git_hashtable_remove(ref->owner->references.packfile, old_name); - - /* Recreate the packed-refs file without the reference */ - error = packed_write(ref->owner); - if (error < GIT_SUCCESS) - goto rename_loose_to_old_name; - - } else { - git__joinpath(old_path, ref->owner->path_repository, old_name); - git__joinpath(new_path, ref->owner->path_repository, ref->name); - - error = gitfo_mv_force(old_path, new_path); - if (error < GIT_SUCCESS) - goto cleanup; - - /* Once succesfully renamed, remove from the cache the reference known by its old name*/ - git_hashtable_remove(ref->owner->references.loose_cache, old_name); - } - - /* Store the renamed reference into the loose ref cache */ - error = git_hashtable_insert(ref->owner->references.loose_cache, ref->name, ref); - - free(old_name); - return error; - -cleanup: - /* restore the old name if this failed */ - free(ref->name); - ref->name = old_name; - return error; - -rename_loose_to_old_name: - /* If we hit this point. Something *bad* happened! Think "Ghostbusters - * crossing the streams" definition of bad. - * Either the packed-refs has been correctly generated and something else - * has gone wrong, or the writing of the new packed-refs has failed, and - * we're stuck with the old one. As a loose ref always takes priority over - * a packed ref, we'll eventually try and rename the generated loose ref to - * its former name. It even that fails, well... we might have lost the reference - * for good. :-/ - */ - - git__joinpath(old_path, ref->owner->path_repository, ref->name); - git__joinpath(new_path, ref->owner->path_repository, old_name); - - /* No error checking. We'll return the initial error */ - gitfo_mv_force(old_path, new_path); - - /* restore the old name */ - free(ref->name); - ref->name = old_name; - - return error; +int git_reference_rename_force(git_reference *ref, const char *new_name) +{ + return git_reference_rename_internal(ref, new_name, 1); } int git_reference_resolve(git_reference **resolved_ref, git_reference *ref) From c7db45e862943e5806db89206a5e3e09c520a9d0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Mon, 28 Mar 2011 14:53:52 +0200 Subject: [PATCH 71/81] Match the comment with the error string MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Carlos Martín Nieto --- include/git2/common.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/git2/common.h b/include/git2/common.h index 7d3518f82..e0c6bc542 100644 --- a/include/git2/common.h +++ b/include/git2/common.h @@ -161,7 +161,7 @@ /** This feature has not been implemented yet */ #define GIT_ENOTIMPLEMENTED (GIT_ERROR - 22) -/** A refernce or object of the same name already exists */ +/** A reference with this name already exists */ #define GIT_EEXISTS (GIT_ERROR - 23) GIT_BEGIN_DECL From 4cd6ed7566b9a1537b667f67b4c1491e3ed70a33 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Mon, 28 Mar 2011 15:05:02 +0200 Subject: [PATCH 72/81] Fix documentation copy error MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Carlos Martín Nieto --- include/git2/refs.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/git2/refs.h b/include/git2/refs.h index 9ecf77295..328a9f1f6 100644 --- a/include/git2/refs.h +++ b/include/git2/refs.h @@ -110,8 +110,8 @@ GIT_EXTERN(int) git_reference_create_oid(git_reference **ref_out, git_repository * Create a new object id reference, overwriting an existing one with * the same name, if it exists. * - * If the new reference isn't a symbolic one, any pointers to the old - * reference become invalid. + * If the new reference isn't an object id one, any pointers to the + * old reference become invalid. * * The reference will be created in the repository and written * to the disk. From 591a9423f53eb9d2baca95d94ebc560667ec8e89 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Mon, 28 Mar 2011 18:40:58 +0200 Subject: [PATCH 73/81] Add tests covering overwriting references MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Carlos Martín Nieto --- tests/t10-refs.c | 98 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) diff --git a/tests/t10-refs.c b/tests/t10-refs.c index 565d636ba..e40536475 100644 --- a/tests/t10-refs.c +++ b/tests/t10-refs.c @@ -295,6 +295,79 @@ BEGIN_TEST(create2, "create a new OID reference") must_pass(gitfo_unlink(ref_path)); /* TODO: replace with git_reference_delete() when available */ END_TEST +static const char *ref_name = "refs/heads/other"; +static const char *ref_master_name = "refs/heads/master"; +BEGIN_TEST(overwrite0, "Overwrite an existing symbolic reference") + git_reference *ref; + git_repository *repo; + + must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); + + must_pass(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); + must_fail(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); + must_pass(git_reference_create_symbolic_force(&ref, repo, ref_name, ref_master_name)); + + git_reference_delete(ref); + git_repository_free(repo); +END_TEST + +BEGIN_TEST(overwrite1, "Overwrite an existing object id reference") + git_reference *ref; + git_repository *repo; + git_oid id; + + must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); + + must_pass(git_reference_lookup(&ref, repo, ref_master_name)); + must_be_true(ref->type & GIT_REF_OID); + git_oid_cpy(&id, git_reference_oid(ref)); + + must_pass(git_reference_create_oid(&ref, repo, ref_name, &id)); + must_fail(git_reference_create_oid(&ref, repo, ref_name, &id)); + must_pass(git_reference_create_oid_force(&ref, repo, ref_name, &id)); + + git_reference_delete(ref); + git_repository_free(repo); +END_TEST + +BEGIN_TEST(overwrite2, "Overwrite an existing object id reference with a symbolic one") + git_reference *ref; + git_repository *repo; + git_oid id; + + must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); + + must_pass(git_reference_lookup(&ref, repo, ref_master_name)); + must_be_true(ref->type & GIT_REF_OID); + git_oid_cpy(&id, git_reference_oid(ref)); + + must_pass(git_reference_create_oid(&ref, repo, ref_name, &id)); + must_fail(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); + must_pass(git_reference_create_symbolic_force(&ref, repo, ref_name, ref_master_name)); + + git_reference_delete(ref); + git_repository_free(repo); +END_TEST + +BEGIN_TEST(overwrite3, "Overwrite an existing symbolic reference with an object id one") + git_reference *ref; + git_repository *repo; + git_oid id; + + must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); + + must_pass(git_reference_lookup(&ref, repo, ref_master_name)); + must_be_true(ref->type & GIT_REF_OID); + git_oid_cpy(&id, git_reference_oid(ref)); + + must_pass(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); + must_fail(git_reference_create_oid(&ref, repo, ref_name, &id)); + must_pass(git_reference_create_oid_force(&ref, repo, ref_name, &id)); + + git_reference_delete(ref); + git_repository_free(repo); +END_TEST + BEGIN_TEST(pack0, "create a packfile for an empty folder") git_repository *repo; char temp_path[GIT_PATH_MAX]; @@ -500,6 +573,25 @@ BEGIN_TEST(rename4, "can not rename a reference with an invalid name") close_temp_repo(repo); END_TEST +BEGIN_TEST(rename5, "can force-rename a reference with the name of an existing reference") + git_reference *looked_up_ref; + git_repository *repo; + + must_pass(open_temp_repo(&repo, REPOSITORY_FOLDER)); + + /* An existing reference... */ + must_pass(git_reference_lookup(&looked_up_ref, repo, packed_head_name)); + + /* Can not be renamed to the name of another existing reference. */ + must_pass(git_reference_rename_force(looked_up_ref, packed_test_head_name)); + + /* Failure to rename it hasn't corrupted its state */ + must_pass(git_reference_lookup(&looked_up_ref, repo, packed_head_name)); + must_be_true(!strcmp(looked_up_ref->name, packed_head_name)); + + close_temp_repo(repo); +END_TEST + BEGIN_TEST(delete0, "deleting a ref which is both packed and loose should remove both tracks in the filesystem") git_reference *looked_up_ref, *another_looked_up_ref; git_repository *repo; @@ -761,6 +853,11 @@ BEGIN_SUITE(refs) ADD_TEST(create1); ADD_TEST(create2); + ADD_TEST(overwrite0); + ADD_TEST(overwrite1); + ADD_TEST(overwrite2); + ADD_TEST(overwrite3); + ADD_TEST(normalize0); ADD_TEST(normalize1); ADD_TEST(normalize2); @@ -773,6 +870,7 @@ BEGIN_SUITE(refs) ADD_TEST(rename2); ADD_TEST(rename3); ADD_TEST(rename4); + ADD_TEST(rename5); ADD_TEST(delete0); ADD_TEST(list0); From dad4a4d582c55394031f8e78a4994da47fca7540 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Tue, 29 Mar 2011 10:47:48 +0200 Subject: [PATCH 74/81] rename: don't return early if the target ref exists MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Carlos Martín Nieto --- src/refs.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/refs.c b/src/refs.c index 20c1d001f..ae745e579 100644 --- a/src/refs.c +++ b/src/refs.c @@ -1050,12 +1050,14 @@ int git_reference_rename_internal(git_reference *ref, const char *new_name, int if (error < GIT_SUCCESS) return error; - /* Ensure we're not going to overwrite an existing reference */ + /* Ensure we're not going to overwrite an existing reference + unless the user has allowed us */ error = git_reference_lookup(&looked_up_ref, ref->owner, new_name); if (error == GIT_SUCCESS && !force) return GIT_EEXISTS; - if (error != GIT_ENOTFOUND) + if (error < GIT_SUCCESS && + error != GIT_ENOTFOUND) return error; From ec99193655606eeb40abc2539296baedaf80522d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Tue, 29 Mar 2011 10:52:29 +0200 Subject: [PATCH 75/81] force-rename test: check for the right name MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Carlos Martín Nieto --- tests/t10-refs.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/t10-refs.c b/tests/t10-refs.c index e40536475..c1511040e 100644 --- a/tests/t10-refs.c +++ b/tests/t10-refs.c @@ -585,9 +585,9 @@ BEGIN_TEST(rename5, "can force-rename a reference with the name of an existing r /* Can not be renamed to the name of another existing reference. */ must_pass(git_reference_rename_force(looked_up_ref, packed_test_head_name)); - /* Failure to rename it hasn't corrupted its state */ - must_pass(git_reference_lookup(&looked_up_ref, repo, packed_head_name)); - must_be_true(!strcmp(looked_up_ref->name, packed_head_name)); + /* Check we actually renamed it */ + must_pass(git_reference_lookup(&looked_up_ref, repo, packed_test_head_name)); + must_be_true(!strcmp(looked_up_ref->name, packed_test_head_name)); close_temp_repo(repo); END_TEST From fc1eeb9dfce7d63141a5b88c62c602e6d0b86cf5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Tue, 29 Mar 2011 11:30:09 +0200 Subject: [PATCH 76/81] Make overwrite test more comprehensive MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Carlos Martín Nieto --- tests/t10-refs.c | 45 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 42 insertions(+), 3 deletions(-) diff --git a/tests/t10-refs.c b/tests/t10-refs.c index c1511040e..528c29ac8 100644 --- a/tests/t10-refs.c +++ b/tests/t10-refs.c @@ -297,17 +297,33 @@ END_TEST static const char *ref_name = "refs/heads/other"; static const char *ref_master_name = "refs/heads/master"; +static const char *ref_branch_name = "refs/heads/branch"; +static const char *ref_test_name = "refs/heads/test"; BEGIN_TEST(overwrite0, "Overwrite an existing symbolic reference") - git_reference *ref; + git_reference *ref, *branch_ref; git_repository *repo; must_pass(git_repository_open(&repo, REPOSITORY_FOLDER)); - must_pass(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); + /* The target needds to exist and we need to check the name has changed */ + must_pass(git_reference_create_symbolic(&branch_ref, repo, ref_branch_name, ref_master_name)); + must_pass(git_reference_create_symbolic(&ref, repo, ref_name, ref_branch_name)); + /* Ensure it points to the right place*/ + must_pass(git_reference_lookup(&ref, repo, ref_name)); + must_be_true(git_reference_type(ref) & GIT_REF_SYMBOLIC); + must_be_true(!strcmp(git_reference_target(ref), ref_branch_name)); + + /* Ensure we can't create it unless we force it to */ must_fail(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); must_pass(git_reference_create_symbolic_force(&ref, repo, ref_name, ref_master_name)); - git_reference_delete(ref); + /* Ensure it points to the right place */ + must_pass(git_reference_lookup(&ref, repo, ref_name)); + must_be_true(git_reference_type(ref) & GIT_REF_SYMBOLIC); + must_be_true(!strcmp(git_reference_target(ref), ref_master_name)); + + must_pass(git_reference_delete(ref)); + must_pass(git_reference_delete(branch_ref)); git_repository_free(repo); END_TEST @@ -322,10 +338,21 @@ BEGIN_TEST(overwrite1, "Overwrite an existing object id reference") must_be_true(ref->type & GIT_REF_OID); git_oid_cpy(&id, git_reference_oid(ref)); + /* Create it */ must_pass(git_reference_create_oid(&ref, repo, ref_name, &id)); + + must_pass(git_reference_lookup(&ref, repo, ref_test_name)); + must_be_true(ref->type & GIT_REF_OID); + git_oid_cpy(&id, git_reference_oid(ref)); + + /* Ensure we can't overwrite unless we force it */ must_fail(git_reference_create_oid(&ref, repo, ref_name, &id)); must_pass(git_reference_create_oid_force(&ref, repo, ref_name, &id)); + /* Ensure it has been overwritten */ + must_pass(git_reference_lookup(&ref, repo, ref_name)); + must_be_true(!git_oid_cmp(&id, git_reference_oid(ref))); + git_reference_delete(ref); git_repository_free(repo); END_TEST @@ -345,6 +372,11 @@ BEGIN_TEST(overwrite2, "Overwrite an existing object id reference with a symboli must_fail(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); must_pass(git_reference_create_symbolic_force(&ref, repo, ref_name, ref_master_name)); + /* Ensure it points to the right place */ + must_pass(git_reference_lookup(&ref, repo, ref_name)); + must_be_true(git_reference_type(ref) & GIT_REF_SYMBOLIC); + must_be_true(!strcmp(git_reference_target(ref), ref_master_name)); + git_reference_delete(ref); git_repository_free(repo); END_TEST @@ -360,10 +392,17 @@ BEGIN_TEST(overwrite3, "Overwrite an existing symbolic reference with an object must_be_true(ref->type & GIT_REF_OID); git_oid_cpy(&id, git_reference_oid(ref)); + /* Create the symbolic ref */ must_pass(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); + /* It shouldn't overwrite unless we tell it to */ must_fail(git_reference_create_oid(&ref, repo, ref_name, &id)); must_pass(git_reference_create_oid_force(&ref, repo, ref_name, &id)); + /* Ensure it points to the right place */ + must_pass(git_reference_lookup(&ref, repo, ref_name)); + must_be_true(git_reference_type(ref) & GIT_REF_OID); + must_be_true(!git_oid_cmp(git_reference_oid(ref), &id)); + git_reference_delete(ref); git_repository_free(repo); END_TEST From 95cde17ca40de7ad7f245c9302f26c5d59f5d873 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 29 Mar 2011 19:22:21 +0300 Subject: [PATCH 77/81] Enforce coding conventions in refs.c Internal methods are static and without the git prefix. 'Force' methods have a `_f` prefix to match the other 'force' methods. --- include/git2/refs.h | 6 +++--- src/refs.c | 29 +++++++++++++++++------------ tests/t10-refs.c | 10 +++++----- 3 files changed, 25 insertions(+), 20 deletions(-) diff --git a/include/git2/refs.h b/include/git2/refs.h index 328a9f1f6..298c66d51 100644 --- a/include/git2/refs.h +++ b/include/git2/refs.h @@ -87,7 +87,7 @@ GIT_EXTERN(int) git_reference_create_symbolic(git_reference **ref_out, git_repos * @param target The target of the reference * @return 0 on success; error code otherwise */ -GIT_EXTERN(int) git_reference_create_symbolic_force(git_reference **ref_out, git_repository *repo, const char *name, const char *target); +GIT_EXTERN(int) git_reference_create_symbolic_f(git_reference **ref_out, git_repository *repo, const char *name, const char *target); /** * Create a new object id reference. @@ -125,7 +125,7 @@ GIT_EXTERN(int) git_reference_create_oid(git_reference **ref_out, git_repository * @param id The object id pointed to by the reference. * @return 0 on success; error code otherwise */ -GIT_EXTERN(int) git_reference_create_oid_force(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id); +GIT_EXTERN(int) git_reference_create_oid_f(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id); /** * Get the OID pointed to by a reference. @@ -243,7 +243,7 @@ GIT_EXTERN(int) git_reference_rename(git_reference *ref, const char *new_name); * and on disk. * */ -GIT_EXTERN(int) git_reference_rename_force(git_reference *ref, const char *new_name); +GIT_EXTERN(int) git_reference_rename_f(git_reference *ref, const char *new_name); /** * Delete an existing reference diff --git a/src/refs.c b/src/refs.c index ae745e579..c6508ca8e 100644 --- a/src/refs.c +++ b/src/refs.c @@ -79,6 +79,11 @@ static int packed_remove_loose(git_repository *repo, git_vector *packing_list); static int packed_sort(const void *a, const void *b); static int packed_write(git_repository *repo); +/* internal helpers */ +static int reference_create_symbolic(git_reference **ref_out, git_repository *repo, const char *name, const char *target, int force); +static int reference_create_oid(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id, int force); +static int reference_rename(git_reference *ref, const char *new_name, int force); + /* name normalization */ static int check_valid_ref_char(char ch); static int normalize_name(char *buffer_out, const char *name, int is_oid_ref); @@ -919,7 +924,7 @@ cleanup: * Internal methods - reference creation *****************************************/ -int git_reference_create_symbolic_internal(git_reference **ref_out, git_repository *repo, const char *name, const char *target, int force) +static int reference_create_symbolic(git_reference **ref_out, git_repository *repo, const char *name, const char *target, int force) { char normalized[MAX_GITDIR_TREE_STRUCTURE_PATH_LENGTH]; int error = GIT_SUCCESS, updated = 0; @@ -976,7 +981,7 @@ cleanup: return error; } -int git_reference_create_oid_internal(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id, int force) +static int reference_create_oid(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id, int force) { int error = GIT_SUCCESS, updated = 0; git_reference *ref = NULL, *old_ref = NULL; @@ -1036,7 +1041,7 @@ cleanup: * We also need to re-insert the reference on its corresponding * in-memory cache, since the caches are indexed by refname. */ -int git_reference_rename_internal(git_reference *ref, const char *new_name, int force) +static int reference_rename(git_reference *ref, const char *new_name, int force) { int error; char *old_name; @@ -1207,22 +1212,22 @@ int git_reference_lookup(git_reference **ref_out, git_repository *repo, const ch int git_reference_create_symbolic(git_reference **ref_out, git_repository *repo, const char *name, const char *target) { - return git_reference_create_symbolic_internal(ref_out, repo, name, target, 0); + return reference_create_symbolic(ref_out, repo, name, target, 0); } -int git_reference_create_symbolic_force(git_reference **ref_out, git_repository *repo, const char *name, const char *target) +int git_reference_create_symbolic_f(git_reference **ref_out, git_repository *repo, const char *name, const char *target) { - return git_reference_create_symbolic_internal(ref_out, repo, name, target, 1); + return reference_create_symbolic(ref_out, repo, name, target, 1); } int git_reference_create_oid(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id) { - return git_reference_create_oid_internal(ref_out, repo, name, id, 0); + return reference_create_oid(ref_out, repo, name, id, 0); } -int git_reference_create_oid_force(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id) +int git_reference_create_oid_f(git_reference **ref_out, git_repository *repo, const char *name, const git_oid *id) { - return git_reference_create_oid_internal(ref_out, repo, name, id, 1); + return reference_create_oid(ref_out, repo, name, id, 1); } /** @@ -1433,12 +1438,12 @@ cleanup: int git_reference_rename(git_reference *ref, const char *new_name) { - return git_reference_rename_internal(ref, new_name, 0); + return reference_rename(ref, new_name, 0); } -int git_reference_rename_force(git_reference *ref, const char *new_name) +int git_reference_rename_f(git_reference *ref, const char *new_name) { - return git_reference_rename_internal(ref, new_name, 1); + return reference_rename(ref, new_name, 1); } int git_reference_resolve(git_reference **resolved_ref, git_reference *ref) diff --git a/tests/t10-refs.c b/tests/t10-refs.c index 528c29ac8..995865dc8 100644 --- a/tests/t10-refs.c +++ b/tests/t10-refs.c @@ -315,7 +315,7 @@ BEGIN_TEST(overwrite0, "Overwrite an existing symbolic reference") /* Ensure we can't create it unless we force it to */ must_fail(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); - must_pass(git_reference_create_symbolic_force(&ref, repo, ref_name, ref_master_name)); + must_pass(git_reference_create_symbolic_f(&ref, repo, ref_name, ref_master_name)); /* Ensure it points to the right place */ must_pass(git_reference_lookup(&ref, repo, ref_name)); @@ -347,7 +347,7 @@ BEGIN_TEST(overwrite1, "Overwrite an existing object id reference") /* Ensure we can't overwrite unless we force it */ must_fail(git_reference_create_oid(&ref, repo, ref_name, &id)); - must_pass(git_reference_create_oid_force(&ref, repo, ref_name, &id)); + must_pass(git_reference_create_oid_f(&ref, repo, ref_name, &id)); /* Ensure it has been overwritten */ must_pass(git_reference_lookup(&ref, repo, ref_name)); @@ -370,7 +370,7 @@ BEGIN_TEST(overwrite2, "Overwrite an existing object id reference with a symboli must_pass(git_reference_create_oid(&ref, repo, ref_name, &id)); must_fail(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); - must_pass(git_reference_create_symbolic_force(&ref, repo, ref_name, ref_master_name)); + must_pass(git_reference_create_symbolic_f(&ref, repo, ref_name, ref_master_name)); /* Ensure it points to the right place */ must_pass(git_reference_lookup(&ref, repo, ref_name)); @@ -396,7 +396,7 @@ BEGIN_TEST(overwrite3, "Overwrite an existing symbolic reference with an object must_pass(git_reference_create_symbolic(&ref, repo, ref_name, ref_master_name)); /* It shouldn't overwrite unless we tell it to */ must_fail(git_reference_create_oid(&ref, repo, ref_name, &id)); - must_pass(git_reference_create_oid_force(&ref, repo, ref_name, &id)); + must_pass(git_reference_create_oid_f(&ref, repo, ref_name, &id)); /* Ensure it points to the right place */ must_pass(git_reference_lookup(&ref, repo, ref_name)); @@ -622,7 +622,7 @@ BEGIN_TEST(rename5, "can force-rename a reference with the name of an existing r must_pass(git_reference_lookup(&looked_up_ref, repo, packed_head_name)); /* Can not be renamed to the name of another existing reference. */ - must_pass(git_reference_rename_force(looked_up_ref, packed_test_head_name)); + must_pass(git_reference_rename_f(looked_up_ref, packed_test_head_name)); /* Check we actually renamed it */ must_pass(git_reference_lookup(&looked_up_ref, repo, packed_test_head_name)); From 9a53df7e0ac364ccba5bd11e9d6c62091f564397 Mon Sep 17 00:00:00 2001 From: Vicent Marti Date: Tue, 29 Mar 2011 19:40:02 +0300 Subject: [PATCH 78/81] refs: Don't allow references to inexistent OIDs --- src/refs.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/src/refs.c b/src/refs.c index c6508ca8e..81706952a 100644 --- a/src/refs.c +++ b/src/refs.c @@ -1319,6 +1319,13 @@ int git_reference_set_oid(git_reference *ref, const git_oid *id) ref_oid = (reference_oid *)ref; + assert(ref->owner); + + /* Don't let the user create references to OIDs that + * don't exist in the ODB */ + if (!git_odb_exists(git_repository_database(ref->owner), id)) + return GIT_ENOTFOUND; + /* duplicate the reference; * this copy will stay on the packfile cache */ if (ref->type & GIT_REF_PACKED) { From 83c95128d924b029b30551f95170df8696583e10 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Thu, 24 Mar 2011 14:15:00 +0100 Subject: [PATCH 79/81] normalize_name: allow more references under refs/ MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Allow any well-formed reference name to live under refs/ removing the condition that they be under refs/{heads,tags,remotes}/ as was the design of git. An exception is made for HEAD which is allowed to contain an OID reference in detached HEAD state. Signed-off-by: Carlos Martín Nieto --- src/refs.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/src/refs.c b/src/refs.c index 81706952a..9661988cc 100644 --- a/src/refs.c +++ b/src/refs.c @@ -1680,8 +1680,9 @@ static int normalize_name(char *buffer_out, const char *name, int is_oid_ref) *buffer_out++ = *current++; } - /* Object id refname have to contain at least one slash */ - if (is_oid_ref && !contains_a_slash) + /* Object id refname have to contain at least one slash, except + * for HEAD in a detached state */ + if (is_oid_ref && !contains_a_slash && strcmp(name, GIT_HEAD_FILE)) return GIT_EINVALIDREFNAME; /* A refname can not end with ".lock" */ @@ -1690,9 +1691,13 @@ static int normalize_name(char *buffer_out, const char *name, int is_oid_ref) *buffer_out = '\0'; - /* For object id references, name has to start with refs/(heads|tags|remotes) */ - if (is_oid_ref && !(!git__prefixcmp(buffer_out_start, GIT_REFS_HEADS_DIR) || - !git__prefixcmp(buffer_out_start, GIT_REFS_TAGS_DIR) || !git__prefixcmp(buffer_out_start, GIT_REFS_REMOTES_DIR))) + /* + * For object id references, name has to start with refs/. Again, + * we need to allow HEAD to be in a detached state. + */ + if (is_oid_ref && + !(git__prefixcmp(buffer_out_start, GIT_REFS_DIR) || + strcmp(buffer_out_start, GIT_HEAD_FILE))) return GIT_EINVALIDREFNAME; return error; From 553fbd640fab5a3f52a20cf1f1ed045ba9349d65 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Carlos=20Mart=C3=ADn=20Nieto?= Date: Tue, 29 Mar 2011 11:43:31 +0200 Subject: [PATCH 80/81] Check for looser reference names MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit res/dummy/a and refs/stash must pass. The other rules are already tested by the rest of the checks. Signed-off-by: Carlos Martín Nieto --- tests/t10-refs.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tests/t10-refs.c b/tests/t10-refs.c index 995865dc8..413811c9d 100644 --- a/tests/t10-refs.c +++ b/tests/t10-refs.c @@ -691,7 +691,8 @@ BEGIN_TEST(normalize0, "normalize a direct (OID) reference name") must_fail(ensure_refname_normalized(OID_REF, "refs/heads/a/", NULL)); must_fail(ensure_refname_normalized(OID_REF, "refs/heads/a.", NULL)); must_fail(ensure_refname_normalized(OID_REF, "refs/heads/a.lock", NULL)); - must_fail(ensure_refname_normalized(OID_REF, "refs/dummy/a", NULL)); + must_pass(ensure_refname_normalized(OID_REF, "refs/dummy/a", NULL)); + must_pass(ensure_refname_normalized(OID_REF, "refs/stash", NULL)); must_pass(ensure_refname_normalized(OID_REF, "refs/tags/a", "refs/tags/a")); must_pass(ensure_refname_normalized(OID_REF, "refs/heads/a/b", "refs/heads/a/b")); must_pass(ensure_refname_normalized(OID_REF, "refs/heads/a./b", "refs/heads/a./b")); From a796d24cf697b0b51aa0ca7ef887e980f0d9fb7a Mon Sep 17 00:00:00 2001 From: Olivier Ramonat Date: Tue, 29 Mar 2011 14:19:40 +0200 Subject: [PATCH 81/81] Fix help message for waf configure --with-sqlite --- wscript | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/wscript b/wscript index b990e148a..f3082aec7 100644 --- a/wscript +++ b/wscript @@ -30,7 +30,7 @@ PPC optimized version (ppc) or the SHA1 functions from OpenSSL (openssl)") opt.add_option('--arch', action='store', default='x86', help='Select target architecture (ia64, x64, x86, x86_amd64, x86_ia64)') opt.add_option('--with-sqlite', action='store_true', default=False, - dest='use_sqlite', help='Disable sqlite support') + dest='use_sqlite', help='Enable sqlite support') opt.add_option('--threadsafe', action='store_true', default=False, help='Make libgit2 thread-safe (requires pthreads)')