The NetBSD Project

CVS log for pkgsrc/textproc/xapian/distinfo-bindings

[BACK] Up to [cvs.NetBSD.org] / pkgsrc / textproc / xapian

Request diff between arbitrary revisions


Default branch: MAIN


Revision 1.18 / (download) - annotate - [select for diffs], Wed Jun 10 17:54:29 2020 UTC (8 weeks ago) by schmonz
Branch: MAIN
CVS Tags: pkgsrc-2020Q2-base, pkgsrc-2020Q2, HEAD
Changes since 1.17: +5 -5 lines
Diff to previous 1.17 (colored)

Update to 1.4.16. From the changelog:

API:

* MSet::snippet(): The snippet now includes trailing punctuation which carries
  meaning or gives useful context.  See
  https://github.com/xapian/xapian/pull/180, reported by Robert Stepanek.

* MSet::snippet(): Fix segfault generating snippet from default-constructed
  MSet.  This probably isn't something you'd typically do, but it shouldn't
  crash.  Found during extended testing of #803 (which only affected git
  master) which was reported by Robert Stepanek.

* Remove trailing full stop from exception messages.  We conventionally don't
  include one, but a few cases didn't follow that convention.

testsuite:

* Replace direct use of ftime() which gives deprecation warnings with recent
  mingw.  Reported by srinivasyadav22.

matcher:

* Fix segfault in rare cases in the query optimiser.  We keep a pointer to the
  most recent posting list to use as a hint for opening the next posting list,
  but the existing mechanism to take ownership of this hint had a flaw.  We now
  invalidate the hint in situations where it might be indirectly deleted which
  is safe, but somewhat conservative.

* Improve the optimisation of an always-matching OP_VALUE_GE to also take
  effect when the value slot's lower bound is equal to the limit of the
  OP_VALUE_GE.  Patch from boda sadalla.

glass backend:

* Report the correct errno value if commit() fails.  We were potentially
  reporting ENOENT from an unlink() call cleaning up a temporary file prior to
  throwing the exception instead.

documentation:

* Fix missing menus in API documentation.  Newer doxygen generates .js files
  which we also need to distribute and install.  Reported by sec^nd on #xapian.

* Note OP_FILTER ignored subquery bug fixed in 1.4.15 as present in 1.4.14 and
  older.

portability:

* Use our own autoconf cache variable namespace (xo_cv_ prefix instead of
  ac_cv_) to avoid colliding with standard autoconf macro use if config.site or
  a shared config.cache is used.  The former case caused a build failure for
  the OpenBSD port with 1.4.15, reported by Lucas R.

* Use clock_gettime() and nanosleep() under modern mingw as these allow higher
  precision than what we previously used.

Bindings:

* Remove code to support SVN snapshots since we stopped using SVN more than 5
  years ago.

* Ignore overloads for logical ops, *, /.  These were already ignored for
  several languages, and aren't actually usefully wrapped for any of the other
  languages.

CSharp:

* Work around mono terminfo parsing bug in more cases.  With this, "make",
  "make check", "make install" and "make uninstall" all work on Ubuntu 18.10.
  Patch from Dipanshu Garg, fixes https://github.com/xapian/xapian/pull/287 and
  #801.

Lua:

* Allow passing a Lua function as a MatchSpy.  This was supposed to be
  supported already, but the typemaps weren't set up.

* On platforms where sizeof(long) is 4, SWIG was wrapping Xapian::BAD_VALUENO
  as a negative constant in Lua, which was then rejected by a check which
  disallows passing negative values for unsigned C++ types.  We now direct SWIG
  to handle Xapian::valueno as double (which is what numbers in Lua usually
  actually are) which gives us an unsigned constant, and also eliminates the
  negative value check.

* Correct documentation - get_description() is wrapped as tostring() in Lua,
  not str() as we previously claimed.

* Add test coverage for passing Lua function for a Stopper.

Perl:

* Resolve the remaining issues and remove the "experimental" marker:

  + Add search_xapian_compat() function which sets up aliases in the
    Search::Xapian namespace to aid writing code which uses either
    Search::Xapian or this module.

  + Allow passing Perl sub for simpler Xapian functor classes.  This fills in a
    missing feature compared to Search::Xapian.  See #523.

  + Remove useless PerlStopper class which was an incomplete copy of the
    apparently non-functional Search::Xapian::PerlStopper.  We now support
    passing a Perl sub for a Stopper object.

  + Adjust some method names to match Search::Xapian.  Iterators now support
    inc() (and dec() where the C++ class supports operator--) like
    Search::Xapian, rather than increment() and prev().  Reported by Eric Wong
    in #523.

  + Drop undocumented and unexpected extra equals() method.

  + Provide compatibility with ENQ_ASCENDING, etc constants.  SWIG wraps these
    as $Xapian::Enquire::ASCENDING, which better matches the C++ API, but
    Search::Xapian wraps this as Search::Xapian::ENQ_ASCENDING, etc so provide
    those too for compatibility.  Reported by Eric Wong in #523.

  + Drop stringification and int conversion overloads.  These seem more
    confusing than helpful, and overloading stringification works badly
    with SWIG-generated bindings.

  + Document remaining known differences from Search::Xapian.

* Update recently tested versions in README.

* Improve documentation.

* Fix t/02pod.t to look for files in right directory.

Ruby:

* Don't print iterator sizes to stdout.  This was some debugging accidentally
  left in as part of a change in 1.4.12.  Patch from Dan Callaghan.

Revision 1.17 / (download) - annotate - [select for diffs], Tue Feb 25 17:55:30 2020 UTC (5 months, 1 week ago) by schmonz
Branch: MAIN
CVS Tags: pkgsrc-2020Q1-base, pkgsrc-2020Q1
Changes since 1.16: +5 -5 lines
Diff to previous 1.16 (colored)

Update to 1.4.15. From the changelog:

API:

* Database::check(): Fix checking of replication changesets.  This reverts a
  change incorrectly made in 1.3.7.

* Database::locked(): Return false instead of true for a closed inmemory DB.

* Database::commit(): If commit() failed with an exception while trying to add
  pending changes (e.g. InvalidArgumentError due to a long term containing zero
  bytes) then a subsequent commit() on the same object would throw the same
  exception.  Now we clear the pending changes in this situation (like we
  already did for failure at other stages in the commit).  This bug remains
  unfixed for the chert backend as it's harder to fix there and the effort to
  fix it and extra risk of breakage don't seem justified for a backend we
  recommend people migrate away from.

* QueryParser::parse_query(): Optimise parsing of multi-word synonyms.

testsuite:

* Use 50-word synonym for qp_scale1 "large" case.  50 divides exactly into the
  number of repetitions we do for the "small" case, which 60 (as used before)
  doesn't.  This makes the two cases a little more comparable and should help
  make this testcase less flaky (see #764).

* Adjust testcase matches1 to work with remote shards where the matcher can
  return slightly better bounds on the number of matches in some cases.
  Resolves 2 XFAILs.

* The testharness get_remote_database() method is now supported for sharded
  databases.  This is needed for keepalive1 to run successfully under multi
  test backends.  Resolves 2 XFAILs of keepalive1.

* Improved test coverage:

  + Test locked() on a closed WritableDatabase, which already returns false (as
    expected) in 1.4.x (but was broken on master).

  + Check multi databases in testsuite - this has been supported by
    Database::check() since 1.4.12.

  + Also test OP_SYNONYM and OP_MAX in emptydb1.

  + Backport testcases boolorbug1, emptynot1, emptymaybe1 and
    phraseweightcheckbug1 from git master - these are regression tests for
    fixed bugs which only affected git master, but it's useful to confirm that
    these bugs don't currently affect 1.4, and ensure they don't get introduced.

* perftest: Store memory sizes as long long since on Microsoft Windows long is
  only 32 bits, which is less than common memory sizes.

matcher:

* Hoist positional check above OP_FILTER.

* Handle OP_FILTER with more than two subqueries correctly.  Previously we'd
  only check the first two subqueries in some situations.

remote backend:

* For a remote WritableDatabase, the client now keeps track of whether there
  are pending changes, and if there aren't then we now do nothing for commit()
  or cancel() calls.  In particular this saves a message exchange when the
  WritableDatabase destructor is called when changes have already been
  committed with an explicit call to commit() (which is what we recommend
  doing, since with an explicit call to commit() you get to see any exception
  which gets thrown).

* When closing a remote prog WritableDatabase, previously an exception could
  leave the remote connection open with the remote server running, and we'd
  then wait for the specified timeout before closing the connection.  Now we
  close the connection before letting the exception propagate.

* Don't swallow exceptions from Database::close() on a remote database.  If
  we aren't in a transaction and so try to commit() and that fails then
  previously the caller would have no indication of the failure.

* Fix handling the reported term weight when remote shards are searched.
  Fixes 5 XFAILs in the testsuite.

* Add missing space to mismatching protocol versions error message.

build system:

* Fix to build when configured with --disable-backend-remote, broken by changes
  in 1.4.14.  Fixes #797, reported by илн алазов.

* The clang and icc compilers both define __GNUC__, which led our ABI mismatch
  message to report them as "g++" with a bogus version (the version of GCC that
  these compilers advertise themselves as, which for clang is always 4.2.0) -
  now we report clang++ or icc along with the actual version of that compiler.

documentation:

* AUTHORS: Apply missed update to the thankyou list for 1.4.14.

* INSTALL: Note that MSVC 2019 works.

* INSTALL: Note that Xapian can use the system uuid.h on AIX and OpenBSD.

portability:

* Simplify probes for snprintf.  The broken snprintf in libbsd in Linux libc4
  is from ~25 years ago so way too ancient to matter now, and all callers
  already handle the pre-ISO semantics of returning -1 for an undersize buffer
  so we don't need to run a test program to probe for this at configure time,
  which is more cross-compile friendly.

* Don't quote messages in #error - the quotes aren't required and appear in the
  compiler output (at least with GCC and clang) making it less readable.

* Use a different approach for getting a 64-bit capable stat() for mingw32.
  This means we now use the same stat variant for mingw32 and MSVC, which
  seems a better plan.

* Work around unhelpful config.status behaviour.  It comments out any #undef
  lines in config.h, even those added via AH_TOP and AH_BOTTOM.  Splitting
  these lines means they don't match the regex hammer config.status uses.

* Avoid -Wdeprecated-copy warnings from clang 10.

* Avoid deprecation warning on recent Linux.  We were including sys/sysctl.h if
  it existed, which it does on Linux but we don't actually use it there.
  Including it now warns that it is deprecated, so skip including it under
  Linux.  Reported on IRC by kumaran.

* Suppress GCC -Wduplicated-branches warning from our API headers in a
  different way which avoids needing a compiler-specific #pragma.

* Workaround closefrom1 failure on macOS.  It seems under macOS our fd tracking
  can end up using fd 10 so start from 13 when testing closefrom() so we don't
  close the fd which our fd tracking is using internally.

debug code:

* Log RemoteConnection::read_at_least() return value.

Revision 1.16 / (download) - annotate - [select for diffs], Tue Dec 17 03:52:58 2019 UTC (7 months, 2 weeks ago) by schmonz
Branch: MAIN
CVS Tags: pkgsrc-2019Q4-base, pkgsrc-2019Q4
Changes since 1.15: +5 -5 lines
Diff to previous 1.15 (colored)

Update to 1.4.14. From the changelog:

API:

* Xapian::QueryParser: Handle "" inside a quoted phrase better.  In a quoted
  boolean term, "" is treated as an escaped ", so handle it in a compatible way
  for quoted phrases.  Previously we'd drop out of the phrase and start a new
  phrase.  Fixes #630, reported by Austin Clements.

* Xapian::Stem: The constructor which takes a stemmer name now takes an
  optional second bool parameter - if this is true, then an unknown stemmer
  name falls back to using the "none" stemmer instead of throwing an exception.
  This allows simply constructing a stemmer from an ISO language code without
  having to worry about whether there's a stemmer for that language, and
  without having to handle an exception if there isn't.

* Xapian::Stem: Fix a bug with handling 4-byte UTF-8 sequences which
  potentially affects most of the stemmers.  None of the stemmers work in
  languages where 4-byte UTF-8 sequences are part of the alphabet, but this
  bug could result in invalid UTF-8 sequences in terms generated from text
  containing high Unicode codepoints such as emoji, which can cause issues (for
  example, in some language bindings).  Fix synced from Snowball git post
  2.0.0.  Reported by Ilari Nieminen in
  https://github.com/snowballstem/snowball/issues/89.

* Xapian::Stem: Add a new is_none() method which tests if this is a "none"
  stemmer.

* Xapian::Weight: The total length of all documents is now made available to
  Xapian::Weight subclasses, and this is now used by DLHWeight, DPHWeight and
  LMWeight.  To maintain ABI compatibility, internally this still fetches the
  average length and the number of documents, multiplies them, then rounds the
  result, but in the next release series this will be handled directly.

* Xapian::Database::locked() on an inmemory database used to always return
  false, but an inmemory Database is always actually a WritableDatabase
  underneath, so now we always report true in this case because it's really
  always report being locked for writing.

* Fix write one past end of std::vector on certain QueryParser parser errors.
  This is undefined behaviour, but the write was always into reserved space, so
  in practice we'd actually get away with it (it was noticed because it
  triggers an error when running under ubsan and using libc++).  Reported by
  Germán M. Bravo.

* MSet::get_matches_estimated(): Improve rounding of result - a bug meant we
  would almost always round down.

* Optimise test for UTF-8 continuation character.  Performing a signed char
  comparison shaves an instruction or two on most architectures.

* Database::get_revision(): Return revision 0 for a Database with no shards
  rather that throwing InvalidOperationError.

* DPHWeight: Avoid dividing by 0 when searching a sharded database when one
  shard is empty.  The result wasn't used in this case, but it's still
  undefined behaviour.  Detected by UBSan.

testsuite:

* Fix failing multi_glass_remoteprog_glass tests on x86.  When the tests are
  run under valgrind, remote servers should be run using the runsrv wrapper
  script, but this wasn't happening for remote servers in multi-databases - now
  it is.  Also, previously runsrv only used valgrind for the remote for an x86
  build that didn't use SSE, but it seems there are x87 instructions in libc
  that are affected by valgrind not providing excess precision, so do this for
  x86 builds which use SSE too.  Together these changes fix failures of
  topercent2, xor2, tradweight1 under backend multi_glass_remoteprog_glass on
  x86.

* Fix C++ One-Definition Rule (ODR) violation in testsuite code.  Two different
  source files linked into apitest were each defining a different `struct
  test`.  Wrap each in an anonymous namespace to localise it to the file it is
  defined and used in.  This was probably harmless in practice, unless trying
  to build with Link-Time Optimisation or similar (which is how it was
  detected).

* Test all language codes in stemlangs1.  The testsuite hardcodes a list of
  supported language codes which hadn't been updated since 2008.

* Improve DateRangeProcessor test coverage.

* The "singlefile" test harness backend manager now creates databases by
  compacting the corresponding underlying backend database (creating it first
  if need be) rather than always creating a temporary database to compact.

* Enable compaction testcases for multi and singlefile test harness backends.

* Add generated database support for remoteprog and remotetcp test harness
  backends.  Implemented by Tanmay Sachan.

* Add test harness support for running testcases using a multi database
  comprised of one local and one remote shard, or two remote shards.
  Implemented by Tanmay Sachan.

* Check if removing existing multi stub failed.  Previously if removing an
  existing stub failed, the test harness would create a temporary new stub and
  then try to rename it over the old one, which will always fail on Microsoft
  Windows.

* Wait for xapian-tcpsrv processes to finish before moving on to the next
  testcase under __WIN32__ like we already do on POSIX platforms.

matcher:

* Handle pruning under a positional check.  This used to be impossible, but
  since 1.4.13 it can happen as we now hoist AND_NOT to just below where we
  hoist the positional checks.  The code on master already handles pruning here
  so this bug is specific to the RELEASE/1.4 branch.  Fixes #796, reported by
  Oliver Runge.

* When searching with collapsing over multiple shards, at least some of which
  are remote, uncollapsed_upper_bound could be too low and
  uncollapsed_lower_bound too high.  This was causing assertion failures in
  testcases msize1 and msize2 under test harness backends
  multi_glass_remoteprog_glass and multi_remoteprog_glass.

* Internally we no longer calculate a bogus total_term_count as the sum of
  total_length * doc_count for all shards.  Instead we just use the sum of
  total_length, which gives the total number of term occurrences.  This change
  should improve the estimated collection_freq values for synonyms.

* Several places where we might divide zero by zero in a database where wdf was
  always zero have been fixed.

* Optimise OP_AND_NOT better.  We now combine its left argument with other
  connected and-like subqueries, and gather up and hoist the negated subqueries
  and apply them together above the combined and-like subqueries, just below
  any positional filters.

* Optimise OP_AND_MAYBE better.  We now combine its left argument with other
  connected and-like subqueries, and gather up and hoist the optional
  subqueries and apply them together above the combined and-like subqueries and
  any hoisted positional filters.

* Treat all BoolWeight queries as scaled by 0 - we can optimise better if we
  know the query is unweighted.

build system:

* configure: Stop using AC_FUNC_MEMCMP.  The autoconf manual marks it as
  "obsolescent", and it seems clear that nobody's relying on it as we're
  missing the "'AC_LIBOBJ' replacement for 'memcmp'" which it would try to
  use if needed.

glass backend:

* Allow zlib compression to reduce size by one byte.  We were specifying an
  output buffer size one byte smaller than the input, but it appears zlib won't
  use the final byte in the buffer, so we actually need to pass the input size
  as the output buffer size.

* Only try to compress Btree item values > 18 bytes, which saves CPU time
  without sacrificing any significant size savings.

remote backend:

* Fix match stats when searching with collapsing over multiple shards and at
  least some shards are remote.  Bug discovered by Tanmay Sachan's test harness
  improvements.

* Ignore orphaned remote protocol replies which can happen when searching with
  a remote shard if an exception is thrown by another shard.  Bug discovered
  by Tanmay Sachan's test harness improvements.

* Wait for xapian-progsrv child to exit when a remote Database or
  WritableDatabase object is closed under __WIN32__ like we already do for
  POSIX platforms.

documentation:

* HACKING: Replace release docs with pointer to the developer guide where they
  are now maintained.

* Correct documentation of initial messages in replication protocol.

tools:

* quest: Report bounds and estimate of number of matches.

* xapian-delve: Improve output when database revision information is not
  available.  We now specially handle the cases of a DB with multiple shards
  and a backend which doesn't support get_revision().

portability:

* Eliminate 2 uses of atoi().  These are potentially problematic in a
  multithreaded application if setlocale() is called by another thread at the
  same time.  See #665.

* Don't check __GNUC__ in visibility.h as the configure probe before defining
  XAPIAN_ENABLE_VISIBILITY checks that the visibility attributes work.  This
  probably makes no difference in practice, as all compilers we're aware of
  which support symbol visibility also define __GNUC__.

* Document Sun C++ requires --disable-shared.  Closes #631.

* Fix warning from GCC 9 with -Wdeprecated-copy (which is enabled by -Wextra)
  if a reference to an Error object is thrown.

* Suppress GCC warning in our API headers when compiling code using Xapian with
  GCC and -Wduplicated-branches.

* Mark some internal classes as final (following GCC -Wsuggest-final-types
  suggestions to allow some method calls to be devirtualised).

* Fix to build with --enable-maintainer-mode and Perl < 5.10, which doesn't
  have the `//=` operator.  It's unlikely developers will have such an old
  Perl, but the mingw environment on appveyor CI does.  The use of `//=` was
  introduced by changes in 1.4.10.

Revision 1.15 / (download) - annotate - [select for diffs], Sat Aug 31 19:31:03 2019 UTC (11 months ago) by schmonz
Branch: MAIN
CVS Tags: pkgsrc-2019Q3-base, pkgsrc-2019Q3
Changes since 1.14: +2 -2 lines
Diff to previous 1.14 (colored)

Fix checksum on new patch.

Revision 1.14 / (download) - annotate - [select for diffs], Thu Aug 29 13:41:15 2019 UTC (11 months, 1 week ago) by schmonz
Branch: MAIN
Changes since 1.13: +2 -1 lines
Diff to previous 1.13 (colored)

Apply previous fix to PHP5 as well.

Revision 1.13 / (download) - annotate - [select for diffs], Wed Aug 28 19:23:38 2019 UTC (11 months, 1 week ago) by schmonz
Branch: MAIN
Changes since 1.12: +2 -1 lines
Diff to previous 1.12 (colored)

Reorder includes in PHP bindings to fix SmartOS bulk build error:
"/usr/include/locale.h:53:12: error: 'lconv' is already declared in this scope"
<https://trac.xapian.org/ticket/793>

Revision 1.12 / (download) - annotate - [select for diffs], Fri Aug 2 21:28:12 2019 UTC (12 months ago) by schmonz
Branch: MAIN
Changes since 1.11: +5 -5 lines
Diff to previous 1.11 (colored)

Update to 1.4.12. From the changelog:

API:

* Xapian::PostingSource: When a PostingSource without a clone() method is used
  with a Database containing multiple shards, the documented behaviour has
  always been that Xapian::InvalidOperationError is thrown.  However, since at
  least 1.4.0, this exception hasn't been thrown, but instead a single
  PostingSource object would get used for all the shards, typically leading to
  incorrect results.  The actual behaviour now matches what was documented.

* Xapian::Database: Add size() method which reports the number of shards.

* Xapian::Database::check(): You can now pass a stub database which will check
  all the databases listed in it (or throw Xapian::UnimplementError for
  backends which don't support checking).

* Xapian::Document: When updating a document use a emplace_hint() to make the
  bulk insertion O(n) instead of O(n·log(n)), and use std::move() to avoid
  copying OmDocumentTerm objects.

* Xapian::Query: Add missing get_unique_terms_end() method.

* Xapian::iterator_valid(): Implement for Utf8Iterator

testsuite:

* Fix keepalive1 failures on some platforms.  On some platforms a timeout
  gives NetworkTimeoutError and on others NetworkError - since 1.4.10 changed
  to checking the exact exception type, keepalive1 has been failing on the
  former set of platforms.  We now just check for NetworkError or a subclass
  here (since NetworkTimeoutError is a subclass of NetworkError).

* Run cursordelbug1 testcase with multi databases too.

matcher:

* Ownership of PostingSource objects during the match now makes use of the
  optional reference-counting mechanism rather than a separate flag.

remote backend:

* Fix remote protocol design bug.  Previously some messages didn't send a reply
  but could result in an exception being sent over the link.  That exception
  would then get read as a response to the next message instead of its actual
  response so we'd be out of step.  Fixes #783, reported by Germán M. Bravo.
  This fix necessitated a minor version bump in the remote protocol (to 39.1).
  If you are upgrading a live system which uses the remote backend, upgrade the
  servers before the clients.

* Fix socket leaks on errors during opening a database.  Fixes
  https://github.com/xapian/xapian/pull/237 and #781, reported by Germán M.
  Bravo.

* Don't close remote DB socket on receiving EOF as the levels above won't
  know it's been closed and may try to perform operations on it, which would be
  problematic if that fd gets reused in the meantime.  Leaving it open means
  any further operations will also get EOF.  Reported by Germán M. Bravo.

* We add a wrapper around the libc socket() function which deals with the
  corner case where SOCK_CLOEXEC is defined but socket() fails if it is
  specified (which can happen with a newer libc and older kernel).
  Unfortunately, this wrapper wasn't checking the returned value from socket()
  correctly, so when SOCK_CLOEXEC was specified and non-zero it would create
  the socket() with SOCK_CLOEXEC, then leak that one and create it again
  without SOCK_CLOEXEC.  We now check the return value properly.

* Fix potential infinite loop in ValueCountMatchSpy::merge_results() if passed
  serialised results with extra data appended (which shouldn't happen in normal
  use).

build system:

* Current versions of valgrind result in false positives on current versions of
  macOS, so on this platform configure now only enables use of valgrind if it's
  specified explicitly.  Fixes #713, reported by Germán M. Bravo.

* Refactor macros to probe for compiler flags so they automatically cache
  their results and consistently report success/failure.

* Rename our custom TYPE_SOCKLEN_T macro to XAPIAN_TYPE_SOCKLEN_T.  The
  AX_TYPE_SOCKLEN_T macro defines an alias of TYPE_SOCKLEN_T for itself which
  means it can get used instead in some situations, but it isn't compatible
  with our macro.  We can't just switch to AX_TYPE_SOCKLEN_T as it doesn't
  handle cases we need, so just rename our macro to avoid potential problems.

documentation:

* Improve API documentation for Xapian::Query class.  Add missing doc
  comments and improve some of the existing ones.  Problems highlighted by
  илн алазов in #790.

* Add Unicode consortium names and codes for categories from Chapter 4, Version
  11 of the Unicode standard.  Patch from David Bremner.

* Improve configure --help output - drop "[default=no]" for --enable-*
  options which default off.  Fixes #791, reported by and patch from илн
  алазов.

* Fix API documentation typo - Query::op (the type) not op_ (a parameter name).

* Note which version Document::remove_postings() was added in.

* In the remote protocol documentation, MSG_REPLACEDOCUMENTTERM was documented
  as not having a reply, but actually REPLY_ADDDOCUMENT is sent.

* Update list of <xapian/iterator.h> users.

tools:

* copydatabase: A change in 1.4.6 which added support for \ as directory
  separator on platforms where that's the norm broke the code in copydatabase
  which removes a trailing slash from input databases.  Bug reported and
  culprit commit identified by Eric Wong.

portability:

* Resolve crash on Windows when using clang-cl and MSVC.  Reported by Christian
  Mollekopf in https://github.com/xapian/xapian/pull/256.

* Add missing '#include <cstring>'.  Patch from Tanmay Sachan.

* Fix str() helper function when converting the most negative value
  of a signed integer type.

* Avoid calling close() on fd we know must actually be a WIN32 SOCKET.

* Include <ios> not <iomanip> for std::boolalpha.

* Rework setenv() compatibility handling.  Now that Solaris 9 is dead we can
  assume setenv() is provided by Unix-like platforms (POSIX requires it).  For
  other platforms, provide a compatibility implementation of setenv() which
  so the compatibility code is encapsulated in one place rather than replicated
  at every use.

* Fix maintainer-mode builds on Solaris where /bin/tr is not POSIX compliant.
  We now use the simple workaround suggested by the autoconf manual.

* Improve support for Sun C++ (see #631):

  + Suppress unhelpful warning for lambda with multiple return statements.

  + Enable reporting the tags corresponding to warnings, which we need
    to know in order to suppress any new unhelpful warnings.

  + Adjust our workaround for bug with this compiler's <cmath> header to avoid
    a compiler warning.

  + Use -xldscope=symbolic for Sun C++.  This flag is roughly equivalent to
    -Bsymbolic-functions which we've probed for with GNU ld since Xapian 1.0.0.

And from the changelog for the language bindings:

Documentation:

* Update bindings HACKING document.  Reported as out of date by Niwesh Gupta.

CSharp:

* Work around mono terminfo parsing bug - older cli-sn fails with e.g.
  TERM=xterm-256color due to: https://github.com/mono/mono/issues/6752
  Encountered on Kubuntu 18.10 and debugged by Tejasvi Tomar.  Seems to be
  fixed in the mono version in Debian buster.

Perl:

* Suppress warnings from older Perl headers due to use of constructs which look
  like C++11 user-defined literals.  They're fixed in newer versions so they're
  just noise in our build.  We were working around these in the CI build, so
  drop that workaround as we want to make the build warning-clean for users
  too.  Reported by daniel93 on #xapian.

Python3:

* Fix build for changes in Sphinx 2.0 (which drops support for sphinx.main()).
  Fixes #778, reported by karolyi.  Also reported by Gaurav Arora.

* We now throw UnicodeEncodeError for bad Unicode string input.  Previously
  cases such as a lone surrogate would be handled by quietly skipping the bad
  codepoints when converting to UTF-8 to pass to Xapian.

* We no longer use the deprecated old-style Py_UNICODE API, which currently
  gives deprecation warnings and is slated to be removed in Python 4.0.

Ruby:

* Add support for block iteration.  All the iterator methods in the Ruby API
  now accept an optional block.  If no block is given an array is returned so
  existing code will still work.  Partly based on a patch in
  https://github.com/xapian/xapian/pull/232 from Cong Ding.

* Add missing wrappers for all the C++ methods returning iterators which
  weren't wrapped for Ruby.  Fixes #777, reported by do.

* Suppress warnings from Ruby 2.3 headers due to use of register and
  constructs which look like C++11 user-defined literals.  They're fixed in
  newer versions so they're just noise in our build.  We were working around
  these in the CI build, so drop that workaround as we want to make the build
  warning-clean for users too.  Reported by daniel93 on #xapian.

* smoketest.rb: Don't leave temporary databases behind in /tmp.

Revision 1.11 / (download) - annotate - [select for diffs], Mon Mar 4 01:38:00 2019 UTC (17 months ago) by schmonz
Branch: MAIN
CVS Tags: pkgsrc-2019Q2-base, pkgsrc-2019Q2, pkgsrc-2019Q1-base, pkgsrc-2019Q1
Changes since 1.10: +5 -5 lines
Diff to previous 1.10 (colored)

Update to 1.4.11. From the changelog:

API:

* MSet::SNIPPET_CJK_NGRAM - new flag for use with MSet::snippet() to enable
  support for selecting and highlighting snippets which works with the
  QueryParser and TermGenerator FLAG_CJK_NGRAM flags.  This mode can also be
  enabled by setting environment variable XAPIAN_CJK_NGRAM to a non-empty
  value.  (There was nominally already support for XAPIAN_CJK_NGRAM in
  MSet::snippet(), but it didn't work usefully - the highlighting added was all
  empty start/end pairs at the end of the span of CJK characters containing the
  CJK ngram terms, which to the user would typically look like it was selecting
  the end of the text and not highlighting anything).

* Deprecate XAPIAN_CJK_NGRAM environment variable.  There are now flags which
  can be used instead in all cases, and there's sadly no portable thread-safe
  way to read an environment variable so checking environment variables is
  problematic in library code that may be used in multithreaded programs.

* Query::OP_ELITE_SET currently incorrectly recursively flattens any OP_OR (or
  OP_OR-like) subqueries into the list of subqueries it selects from - until
  that's fixed, we now select from the full exploded list rather than the last
  n (where n is the number of direct subqueries of the OP_ELITE_SET).

testsuite:

* Testcases which need a generated database now get run with a sharded
  database.

* Avoid using strerror() in the testsuite which removes an obstacle to running
  tests in parallel in separate threads.

matcher:

* Extend detection of cases of OP_SYNONYM with disjoint subqueries (which means
  we don't need document length) which was added in 1.4.8 - we now detect when
  all subqueries are different terms, or when all subqueries are
  non-overlapping wildcards.  The second case is what QueryParser produces for
  a wildcard or partial query with a query prefix which maps to more than one
  term prefix.

glass backend:

* Handle an empty value slot lower bound gracefully.  This shouldn't happen for
  a non-empty slot, but has been reported by a notmuch user so it seems there
  is (or perhaps was as the database was several years old) a way it can come
  about.  We now check for this situation and set the smallest possible valid
  lower bound instead, so other code assuming a valid lower bound will work
  correctly.  Reported by jb55.

chert backend:

* Handle an empty value slot lower bound gracefully, equivalent to the change
  made for glass.

documentation:

* HACKING: We no longer use auto_ptr<>.

* NEWS: Correct factual error in old entry - the 0.4.1 release was Open Muscat
  not OmSee (the OmSee name was only applied after that final release was made,
  and only used internally to BrightStation).

portability:

* Suppress more clang -Wself-assign-overloaded warnings in testcases which are
  deliberately testing handling of self-assignment.

* Add missing includes of <cerrno>.  Fixes #776, reported by Matthieu Gautier.

debug code:

* When configured with --enable-log, the O_SYNC flag was always specified when
  opening the logfile, with the intention that the most recent log entries
  wouldn't get lost if there was a crash, but O_SYNC can incur a significant
  performance overhead and most debugging is not of such crashes.  So we no
  longer specify O_SYNC by default, but you can now request synchronous logging
  by including %! anywhere in the filename specified with XAPIAN_DEBUG_LOG
  (the %! is replaced with the empty string).  We also now use O_DSYNC if
  available in preference to O_SYNC, since the mtime of the log file isn't
  important.

Revision 1.10 / (download) - annotate - [select for diffs], Tue Feb 12 19:21:18 2019 UTC (17 months, 3 weeks ago) by schmonz
Branch: MAIN
Changes since 1.9: +5 -5 lines
Diff to previous 1.9 (colored)

Xapian-core 1.4.10 (2019-02-12):

API:

* DatabaseClosedError: New exception class thrown instead of DatabaseError when
  an operation is attempted which can't be completed because it involves a
  database which close() was previously called on.  DatabaseClosedError is a
  subclass of DatabaseError so existing code catching DatabaseError will still
  work as before.  Fixes #772, reported by Germán M. Bravo.  Patch from
  Vaibhav Kansagara.

* DatabaseNotFoundError: New exception class thrown instead of
  DatabaseOpeningError when the problem is the problem is "file not found" or
  similar.  DatabaseNotFoundError is a subclass of DatabaseOpeningError so
  existing code catching DatabaseOpeningError will still work as before.  Fixes
  #773, reported by Germán M. Bravo.  Patch from Vaibhav Kansagara.

* Query: Make &=, |= and ^= on Query objects opportunistically append to
  an existing query with a matching query operator which has a reference
  count of 1.  This provides an easy way to incrementally build flatter query
  trees.

* Query: Support `query &= ~query2` better - this now is handled exactly
  equivalent to `query = query & ~query2` and gives `query AND_NOT query2`
  instead of `query AND (<alldocuments> AND_NOT query2)`.

* QueryParser: Now uses &=, |= and ^= to produce flatter query trees.  This
  fixes problems with running out of stack space when handling Query object
  trees built by abusing QueryParser to parse very large machine-generated
  queries.

* Stopper: Fix incorrect accents in Hungarian stopword list.  Patch from David
  Corbett.

testsuite:

* Test MSet::snippet() with small and zero lengths.  Fixes #759.  Patch from
  Vaibhav Kansagara.

* Fix testcase stubdb4 annotations - this testcase doesn't need a backend.

* Add PATH annotation for testcases needing get_database_path() to avoid having
  to repeatedly list the backends where this is supported in testcase
  annotations.

* TEST_EXCEPTION helper macro now checks that the exact specified exception
  type is thrown.  Previously it would allow a subclass of the specified
  exception type, but in testcases we really want to be able to test for an
  exact type.  Issue noted by Vaibhav Kansagara on IRC.

matcher:

* Map OP_VALUE_GE/OP_VALUE_LE on an empty slot to EmptyPostList.  We already do
  this for OP_VALUE_RANGE, and it's a little more efficient than creating a
  postlist object which checks the empty value slot.

glass backend:

* We no longer flush all pending positional changes when a postlist, termlist
  or all-terms is opened on a modified WritableDatabase.  Doing so was
  incurring a significant performance cost, and the first of these happens
  internally when `replace_document(term, doc)` is used, which is the usual way
  to support non-numeric unique ids.  We now only flush pending positional
  changes when committing.  Reported and diagnosed by Germán M. Bravo.

remote backend:

* Use poll() where available instead of select().  poll() is specified by
  POSIX.1-2001 so should be widely available by now, and it allows watching any
  fd (select() is limited to watching fds < FD_SETSIZE).  For any platforms
  which still lack poll() we now workaround this select() limitation when a
  high numbered fd needs to be watched (for example, by trying a non-blocking
  read or write and on EAGAIN sleeping for a bit before retrying).

* Stop watching fds for "exceptional conditions" - none of these are relevant
  to our usage.

* Remove 0.1s timeout in ready_to_read().  The comment says this is to avoid a
  busy loop, but that's out of date - the matcher first checks which remotes
  are ready to read and then does a second pass to handle those which weren't
  with a blocking read.

build system:

* Stop probing for header sys/errno.h which is no longer used - it was only
  needed for Compaq C++, support for which was dropped in 1.4.8.

documentation:

* docs/valueranges.html: Update to document RangeProcessor instead of
  ValueRangeProcessor - the latter is deprecated and will be gone in the next
  release series.

* Document RangeProcessor::operator()() returns OP_INVALID to signal it doesn't
  recognise a range.

* Update some URLs for pages which have moved.

* Use https for URLs where available.

* HACKING: Update "empty()" section for changes in C++11.

portability:

* Suppress clang warnings for self-assignment tests.  Some testcases trigger
  this new-ish clang warning while testing that self-assignment works, which
  seems a useful thing to be testing - at least one of these is a regression
  test.

* Add std::move to fix clang -Wreturn-std-move warning (which is enabled by
  -Wall).

* Add casts to fix ubsan warnings.  These cases aren't undefined behaviour, but
  are reported by ubsan extra checks implicit-integer-truncation and/or
  implicit-conversion which it is useful to be able to enable to catch
  potential bugs.

* Fix check for when to use _byteswap_ulong() - in practice this would only
  have caused a problem if a platform provided _byteswap_ushort() but not
  _byteswap_ulong(), but we're not aware of any which do.

* Fix return values of do_bswap() helpers to match parameter types (previously
  we always returned int and only supported swapping types up to 32 bits, so
  this probably doesn't result in any behavioural changes).

* Only include <intrin.h> if we'll use it instead of always including it when
  it exists.  Including <intrin.h> can result in warnings about duplicate
  declarations of builtin functions under mingw.

* Remove call to close()/closesocket() when the argument is always -1 (since
  the change to use getaddrinfo() in 1.3.3).

Revision 1.9 / (download) - annotate - [select for diffs], Mon Nov 5 05:42:46 2018 UTC (21 months ago) by schmonz
Branch: MAIN
CVS Tags: pkgsrc-2018Q4-base, pkgsrc-2018Q4
Changes since 1.8: +5 -5 lines
Diff to previous 1.8 (colored)

Update to 1.4.9. From the changelog:

API:

* Document::add_posting(): Fix bugs with the change in 1.4.8 to more
  efficiently handle insertion of a batch of extra positions in ascending
  order.  These could lead to missing positions and corrupted encoded
  positional data.

remote backend:

* Avoid hang if remote connection shutdown fails by not waiting for the
  connection to close in this situation.  Seems to fix occasional hangs seen on
  macOS.  Patch from Germán M. Bravo.

Revision 1.8 / (download) - annotate - [select for diffs], Sun Oct 28 03:43:17 2018 UTC (21 months, 1 week ago) by schmonz
Branch: MAIN
Changes since 1.7: +5 -5 lines
Diff to previous 1.7 (colored)

Update to 1.4.8. From the changelog:

API:

* QueryParser,TermGenerator: Add new stemming mode STEM_SOME_FULL_POS.
  This stores positional information for both stemmed and unstemmed terms,
  allowing NEAR and ADJ to work with stemmed terms.  The extra positional
  information is likely to take up a significant amount of extra disk space so
  the default STEM_SOME is likely to be a better choice for most users.

* Database::check(): Fetch and decompress the document data to catch problems
  with the splitting of large data into multiple entries, corruption of the
  compressed data, etc.  Also check that empty document data isn't explicitly
  stored for glass.

* Fix an incorrect type being used for term positions in the TermGenerator API.
  These were Xapian::termcount but should be Xapian::termpos.  Both are
  typedefs for the same 32-bit unsigned integer type by default (almost always
  "unsigned int") so this change is entirely compatible, except that if you
  were configuring 1.4.7 or earlier with --enable-64bit-termcount you need to
  also use the new --enable-64bit-termpos configure option with 1.4.8 and up or
  rebuild your applications.  This change was necessary to make
  --enable-64bit-termpos actually useful.

* Add Document::remove_postings() method which removes all postings in a
  specified term position range much more efficiently than by calling
  remove_posting() repeatedly.  It returns the number of postings removed.

* Fix bugs with handling term positions >= 0x80000000.  Reported by Gaurav
  Arora.

* Document::add_posting(): More efficiently handle insertion of a batch of
  extra positions in ascending order.

* Query: Simplify OP_SYNONYM with single OP_WILDCARD subquery by converting to
  OP_WILDCARD with combiner OP_SYNONYM, which means such cases can take
  advantage of the new matcher optimisation in this release to avoid needing
  document length for OP_WILDCARD with combiner OP_SYNONYM.

matcher:

* Avoid needing document length for an OP_WILDCARD with combiner OP_SYNONYM.
  We know that we can't get any duplicate terms in the expansion of a wildcard
  so the sum of the wdf from them can't possibly exceed the document length.

* OP_SYNONYM: No longer tries to initialise weights for its subquery, which
  should reduce the time taken to set up a large wildcard query.

* OP_SYNONYM: Fix frequency estimates when OP_SYNONYM is used with a
  subquery containing OP_XOR or OP_MAX - in such cases the frequency
  estimates for the first subquery of the OP_XOR/OP_MAX were used for
  all its subqueries.  Also the estimated collection frequency is
  now rounded to the nearest integer rather than always being rounded
  down.

glass backend:

* Revert change made in 1.4.6:

    Enable glass's "open_nearby_postlist" optimisation (which especially helps
    large wildcard queries) for writable databases without any uncommitted
    changes as well.

  The amended check isn't conservative enough as there may be postlist changes
  in the inverter while the table is unmodified.  This breaks testcase
  T150-tagging.sh in notmuch's testsuite, reported by David Bremner.

* When indexing a document without any terms we now avoid some unnecessary work
  when storing its termlist.

tools:

* xapian-delve: Test for all docs empty using get_total_length() which is
  slightly simpler internally than get_avlength(), and avoids an exact floating
  point equality check.

examples:

* quest: Support --weight=coord.

* xapian-pos: New tool to show term position info to help debugging when using
  positional information in more complex ways.

portability:

* Fix undefined behaviour from C++ ODR violation due to using the same name
  two different non-static inline functions.  It seems that with current GCC
  versions the desired function always ends up being used, but with current
  clang the other function is sometimes used, resulting in database corruption
  when using value slots in docid 16384 or higher with the default glass
  backend.  Patch from Germán M. Bravo.

* Suppress alignment cast warning on sparc Linux.  The pointer being cast is to
  a record returned by getdirentries(), so it should be suitable aligned.

* Drop special handling for Compaq C++.  We never actually achieved a working
  build using it, and I can find no evidence that this compiler still exists,
  let alone that it was updated for C++11 which we now require.

* Create new database directories in race-free way.

* Avoid throwing and handling an exception in replace_document() when
  adding a document with a specified docid which is <= last_docid but currently
  unused.

* Use our portable code for handling UUIDs on all platforms, and only use
  platform-specific code for generating a new UUID.  This fixes a bug with
  converting UUIDs to and from string representation on FreeBSD, NetBSD and
  OpenBSD on little-endian platforms which resulted in reversed byte order in
  the first three components, so the same database would report a different
  UUID on these platforms compared to other platforms.  With this fix, the
  UUIDs of existing databases will appear to change on these platforms
  (except in rare "palindronic" cases).  Reported by Germán M. Bravo.

* Fix to build with a C++17 compiler.  Previously we used a "byte" type
  internally which clashed with "std::byte" in source files which use
  "using namespace std;".  Fixes #768, reported by Laurent Stacul.

* Adjust apitest testcase stubdb2 to allow for NetBSD oddity: NetBSD's
  getaddrinfo() in IPv4 mode seems to resolve ::1 to an IPv4 address on the
  local network.

* Avoid timer_create() on OpenBSD and NetBSD.  On OpenBSD it always fails with
  ENOSYS (and there's no prototype in the libc headers), while on NetBSD it
  seems to work, but the timer never seems to fire, so it's useless to us (see
  #770).

* Use SOCK_NONBLOCK if available to avoid a call to fcntl().  It's supported by
  at least Linux, FreeBSD, NetBSD and OpenBSD.

* Use O_NOINHERIT for O_CLOEXEC on Windows.  This flag has essentially the same
  effect, and it's common in other codebases to do this.

* On AIX O_CLOEXEC may be a 64-bit constant which won't fit in an int.  To
  workaround this stupidity we now call the non-standard open64x() instead
  of open() when the flags don't fit in an int.

* Add functions to add/multiply with overflow check.  These are implemented
  with compiler builtins or equivalent where possible, so the overflow check
  will typically just require a check of the processor's overflow or carry
  flag.

Revision 1.7 / (download) - annotate - [select for diffs], Sun Aug 26 13:25:33 2018 UTC (23 months, 1 week ago) by schmonz
Branch: MAIN
CVS Tags: pkgsrc-2018Q3-base, pkgsrc-2018Q3
Changes since 1.6: +5 -5 lines
Diff to previous 1.6 (colored)

Update to 1.4.7. From the changelog:

API:

* Database::check(): Fix bogus error reports for documents with length zero
  due to a new check added in 1.4.6 that the doclength was between the stored
  upper and lower bounds, which failed to allow for the lower bound ignoring
  documents with length zero (since documents indexed only by boolean terms
  aren't involved in weighted searches).  Reported by David Bremner.

* Query: Use of Query::MatchAll in multithreaded code causes problems because
  the reference counting gets messed up by concurrent updates.  Document that
  Query(string()) should be used instead of MatchAll in multithreaded code, and
  avoid using it in library code.  Reported by Germán M. Bravo.

* Stem:

  + Stemming algorithms added for Irish, Lithuanian, Nepali and Tamil.

  + Merge Snowball compiler changes which improve code generation.

  + Merge optimisations to the Arabic and Turkish stemmers.

glass backend:

* A long-lived cursor on a table in a WritableDatabase could get into
  an invalid state, which typically resulted in a DatabaseCorruptError
  being thrown with the message:

      Db block overwritten - are there multiple writers?

  But in fact the on-disk database is not corrupted - it's just that
  the cursor in memory has got into an inconsistent state.  It looks
  like we'll always detect the inconsistency before it can cause on-disk
  corruption but it's hard to be completely certain.

  The bug is in code to rebuild the cursor when the underlying table
  changes in ways which require that, which is a fairly rare occurrence
  to start with, and only triggers when a block in the cursor has been
  released, reallocated, and we tried to load it in the cursor at the
  same level - the cursor wrongly assumes it has the current version
  of the block.

  Reported with a reproducer by Sylvain Taverne.  Confirmed by David
  Bremner as also fixing a problem in notmuch for which he hadn't managed
  to find a reduced reproducer.

Revision 1.6 / (download) - annotate - [select for diffs], Fri Jul 6 16:21:56 2018 UTC (2 years, 1 month ago) by schmonz
Branch: MAIN
Changes since 1.5: +5 -5 lines
Diff to previous 1.5 (colored)

Update to 1.4.6. From the changelog:

API:

* API classes now support C++11 move semantics when using a compiler which
  we are confident supports them (currently compilers which define
  __cplusplus >= 201103 plus a special check for MSVC 2015 or later).
  C++11 move semantics provide a clean and efficient way for threaded code to
  hand-off Xapian objects to worker threads, but in this case it's very
  unhelpful for availability of these semantics to vary by compiler as it
  quietly leads to a build with non-threadsafe behaviour.  To address this,
  user code can #define XAPIAN_MOVE_SEMANTICS before #include <xapian.h> to
  force this on, and will then get a compilation failure if the compiler lacks
  suitable support.

* MSet::snippet():

  + We were only escaping output for HTML/XML in some cases, which would
    potentially allow HTML to be injected into output (this has been assigned
    CVE-2018-0499).

  + Include certain leading non-word characters in snippets.  Previously we
    started the snippet at the start of the first actual word, but there are
    various cases where including non-word characters in front of the actual
    word adds useful context or otherwise aids comprehension.  Reported by
    Robert Stepanek in https://github.com/xapian/xapian/pull/180

* Add MSetIterator::get_sort_key() method.  The sort key has always been
  available internally, but wasn't exposed via the public API before, which
  seems like an oversight as the collapse key has long been available.
  Reported by 张on xapian-discuss.

* Database::compact():

  + Allow Compactor::resolve_duplicate_metadata() implementations to delete
    entries.  Previously if an implementation returned an empty string this
    would result in a user meta-data entry with an empty value, which isn't
    normally achievable (empty meta-data values aren't stored), and so will
    cause odd behaviour.  We now handle an empty returned value by interpreting
    it in the natural way - it means that the merged result is to not set a
    value for that key in the output database.

  + Since 1.3.5 compacting a WritableDatabase with uncommitted changes throws
    Xapian::InvalidOperationError when compacting to a single-file glass
    database.  This release adds similar checks for chert and when compacting
    to a multiple-file glass database.

  + In the unlikely event that the total number of documents or the total
    length of all documents overflow when trying to compact a multi-database,
    we throw an exception.  This is now a DatabaseError exception instead of a
    const char* exception (a hang-over from before this code was turned into a
    public API in the library).

* Document::remove_term(): Handle removing term at current TermIterator
  position - previously the underlying iterator was invalidated, leading to
  undefined behaviour (typically a segmentation fault).  Reported by Gaurav
  Arora.

* TermIterator::get_termfreq() now always returns an exact answer.  Previously
  for multi-databases we approximated the result, which is probably either a
  hang-over from when this method was used during Enquire::get_eset(), or else
  due to a thinking that this method would be used in that situation (it
  certainly is not now).  If the user creates a TermIterator object and asks it
  for term frequencies then we really should give them the correct answer - it
  isn't hugely costly and the documentation doesn't warn that it might be
  approximated.

* QueryParser::parse_query():

  + Now adds a colon after the prefix when prefixing a boolean term which
    starts with a colon.  This means the mapping is reversible, and matches
    what omega actually does in this case when it tries to reverse the mapping.
    Thanks to Andy Chilton for pointing out this corner case.

  + The parser now makes use of newer features in the lemon parser generator to
    make parsing faster and use less memory.

* Enquire::get_mset(): Fix bug with get_mset(0, 0, X) when X > 0 which was
  causing an attempt to access an element in an empty vector.  Reported by
  sielicki in #xapian.

* Stem:

  + Add Indonesian stemming algorithm.

  + Small optimisations to almost all stemming algorithms.

* Stopper:

  + Add Indonesian stopword list.

  + The installed version of the Finnish stopword list now has one word per
    line.  Previously it had several space-separated words on some lines, which
    works with C++'s std::istream_iterator but may be inconvenient for use from
    some other languages.

  + The installed versions of stopword lists are now sorted in byte order
    rather than whatever collation order is specified by LC_COLLATE or similar
    at build time.  This makes the build more reproducible, and also may be
    more efficient for loading into some data structures.

* WritableDatabase::replace_document(term, doc): Check for last_docid wrapping
  when used on a sharded database.

* Database::locked(): Consistently throw FeatureUnavailableError on platforms
  where we can't test for a database lock without trying to take it.
  Previously GNU Hurd threw DatabaseLockError while platforms where we don't
  use fcntl() locking at all threw UnimplementedError.

* Database and WritableDatabase constructors: Fix handling of entries for
  disabled backends in stub database files to throw FeatureUnavailableError
  instead of DatabaseError.

* Database::get_value_lower_bound() now works correctly for sharded databases.
  Previously it returned the empty string if any shard had no values in the
  specified slot.

* PostingIterator was failing to keep an internal reference to the parent
  Database object for sharded databases.

* ValueIterator::skip_to() and check() had an off-by-one error in their docid
  calculations in some cases with sharded databases.

* Add Database::get_total_length() method.  Previously you had to calculate
  this from get_avlength() and get_doccount(), taking into account rounding
  issues.  But even then you couldn't reliably get the exact value when total
  length is large since a double's mantissa has more limited precision than an
  unsigned long long.

* Add Xapian::iterator_rewound() for bidirectional iterators, to test if the
  iterator is at the start (useful for testing whether we're done when
  iterating backwards).

* DatabaseOpeningError exceptions now provide errno via get_error_string()
  rather than turning it into a string and including it in the exception
  message.

* WritableDatabase::replace_document(): when passed a Document object which
  came from a database and has unmodified values, we used to always read
  those values into a memory structure.  Now we only do this if the document
  is being replaced to the same document ID which it came from, which should
  make other cases a bit more efficient.

* Enquire::get_eset(): When approximating term frequencies we now round to the
  nearest integer - previously we always rounded down.
matcher:

* OP_VALUE_*: When a value slot's lower and upper bound are equal, we know
  that exactly how many documents the subquery can match (either 0 or those
  bounds).  This also avoids a division by zero which previously happened
  when trying to calculate the estimate.

* Speed up sorting by keys.  Use string::compare() to avoid having to call
  operator< if operator> returns false.

* Fix clamping of maxitems argument to get_mset() - it was being clamped
  to db.get_doccount(), now it's clamped to db.get_doccount() - first.  In
  practice this doesn't actually seem to cause any issues.

* If a match time limit is in effect, when it expires we now clamp
  check_at_least to first + maxitems instead of to maxitems.  In practice this
  also doesn't seem to actually cause any issues (at least we've failed to
  construct a testcase where it actually makes an observable difference).

* Fix percentages when only some shards have positions.  If the final shard
  didn't have positions this would lead to under-counting the total number leaf
  of subqueries which would lead to incorrect positional calculations (and a
  division by zero if the top level of the query was positional.  This bug was
  introduced in 1.4.3.

* OP_NEAR: Fix "phantom positions", where OP_NEAR would think a term without
  positional information occurred at position 1 if it had the lowest term
  frequency amongst the OP_NEAR's subqueries.

* Fix termfreq used in weight calculations for a term occurring more than once
  in the query.  Previously the termfreq for such terms was multiplied by the
  number of different query positions they appeared at.

* OP_SYNONYM: We use the doclength upper bound for the wdf upper bound of a
  synonym - now we avoid fetching it twice when the doclength upper bound is
  explicitly needed.

* Short-cut init() when factor is 0 in most Weight subclasses.  This indicates
  the object is for the term-independent weight contribution, which is always 0
  for most schemes, so there's no point fetching any stats or doing any
  calculations.  This fixes a divide by zero for TfIdfWeight, detected by
  UBSan.

* OP_OR: Fix bug which caused orcheck1 to fail once hooked up to run with the
  inmemory backend.

* Iterating of positions has been sped up, which means phrase matching is now
  faster (by a little over 5% in some simple tests).

* Fix use after free of QueryOptimiser hint in certain cases involving
  multiple databases only some of which have positional information.
  This bug was introduced by changes in xapian-core 1.4.3.  Fixes #752,
  reported and analysed by Robert Stepanek.

* An unweighted OP_AND_MAYBE is now optimised to just its left branch - the
  other branch or branches only contribute weight, so can be completely ignored
  when the operator is unweighted.

glass backend:

* Fix glass freelist bug when changes to a new database which didn't modify the
  termlist table were committed.  In this corner case, a block which had been
  allocated to be the root block in the termlist table was leaked.  This was
  largely harmless, except that it was detected by Database::check() and caused
  it to report an error.  Reported by Antoine Beaupré and David Bremner.

* Fix glass freelist bug with cancel_transaction().  The freelist wasn't
  reset to how it was before the transaction, resulting in leaked blocks.
  This was largely harmless, except that it was detected by Database::check()
  and caused it to report an error.

* Improve the per-term wdf upper bound.  Previously we used min(cf(term),
  wdf_upper_bound(db)) which is tight for any terms which attain that
  upper bound, and also for terms with termfreq == 1 (the latter are common
  in the database (e.g. 66% for a database of wikipedia), but probably
  much less common in searches).  When termfreq > 1 we now use
  max(first_wdf(term), cf(term) - first_wdf(term)), which means terms with
  termfreq == 2 will also attain their bound (another 11% for the same
  database) while terms with higher termfreq but below the global bound will
  get a tighter bound.

* Fix Database::locked() on single-file glass db to just return false (such
  databases can't be opened as a WritableDatabase so there can't be a write
  lock).  Previously this failed with: "DatabaseLockError: Unable to get write
  lock on /flintlock: Testing lock"

* Fix compaction when both the input and output are specified as a file
  descriptor.  Previously this threw an exception due to an overeager check
  that destination != source.

* Use O_TRUNC when compacting to single file.  If the output already exists but
  is larger than our output we don't want to just overwrite the start of it.
  This case also used to result in confusing compaction percentages.

* Enable glass's "open_nearby_postlist" optimisation (which especially helps
  large wildcard queries) for writable databases without any uncommitted
  changes as well.

* Make get_unique_terms() more efficient for glass.  We approximate
  get_unique_terms() by the length of the termlist (which counts boolean terms
  too) but clamp this to be no larger than the document length.  Since we need
  to open the termlist to get its length, it makes more sense to get the
  document length from that termlist for no extra cost rather than looking it
  up in the postlist table.

* Database::check() now checks document lengths against the stored document
  length lower and upper bounds.  Patch from Uppinder Chugh.  Fixes
  https://trac.xapian.org/ticket/617.

* Fix bogus handling of most-recently-read value slot statistics.  It seems
  that we get lucky and this can't actually cause a problem in practice due
  to another layer of caching above, but if nothing else it's a bug waiting to
  happen.

* If we fail to create the directory for a new database because the path
  already exists, the exception now reports EEXIST as the errno value rather
  than whatever errno value happened to be set from an earlier library call.

remote backend:

* xapian-tcpsrv --one-shot no longer forks.  We need fork to handle multiple
  concurrent connections, but when handling a single connection forking just
  adds overhead and potentially complicates process management for our caller.
  This aligns with the behaviour under __WIN32__ where we use threads instead
  of forking, and service the connection from the main thread with --one-shot.

* Fix repeat call to ValueIterator::check() on the same docid to not always
  set valid to true for remote backend.

inmemory backend:

* Fix repeat call to ValueIterator::check() on the same docid to not always
  set valid to true for inmemory backend.

* Use binary chop instead of linear search in all places where we're searching
  for a term or document - we weren't taking advantage of the sorted order
  everywhere.

tools:

* xapian-delve:

  + Document values can contain binary data, so escape them by default for
    output.  Other options now supported are to decode as a packed integer
    (like omindex uses for last modified), decode using
    Xapian::sortable_unserialise(), and to show the raw form (which was the
    previous behaviour).

  + Report current database revision.

* xapian-inspect:

  + Report entry count when opening table

  + Support inspecting single file DBs via a new --table option (which can also
    be used with a non-single-file DB instead of specifying the path to the
    table).

  + Add "first" and "last" commands which jump to the first/last entry in the
    current table respectively.

  + "until" now counts and reports the number of entries advanced by.

  + Document "until" with no arguments - this advances to the end of the table,
    but wasn't mentioned in the help.

  + Commands "goto" and "until" which take a key as an argument now expect the
    key in the same escaped form that's used for display.  This makes it much
    simpler to interact with tables with binary keys.

  + Fix to expect .glass not .DB extension of glass tables.

Revision 1.5 / (download) - annotate - [select for diffs], Mon Aug 21 18:49:39 2017 UTC (2 years, 11 months ago) by schmonz
Branch: MAIN
CVS Tags: pkgsrc-2018Q2-base, pkgsrc-2018Q2, pkgsrc-2018Q1-base, pkgsrc-2018Q1, pkgsrc-2017Q4-base, pkgsrc-2017Q4, pkgsrc-2017Q3-base, pkgsrc-2017Q3
Changes since 1.4: +2 -1 lines
Diff to previous 1.4 (colored)

Add patchsum for previous.

Revision 1.4 / (download) - annotate - [select for diffs], Mon Aug 7 22:24:05 2017 UTC (2 years, 11 months ago) by schmonz
Branch: MAIN
Changes since 1.3: +2 -2 lines
Diff to previous 1.3 (colored)

Link directly with libtclstub.a (there's no .so). Fixes build on NetBSD,
doesn't break build on OS X.

Revision 1.3 / (download) - annotate - [select for diffs], Mon Aug 7 20:01:39 2017 UTC (2 years, 11 months ago) by schmonz
Branch: MAIN
Changes since 1.2: +2 -1 lines
Diff to previous 1.2 (colored)

Allow installing multiple versions of Lua bindings. Bump PKGREVISION.

Revision 1.2 / (download) - annotate - [select for diffs], Wed Aug 2 04:42:23 2017 UTC (3 years ago) by schmonz
Branch: MAIN
Changes since 1.1: +2 -1 lines
Diff to previous 1.1 (colored)

Specify path to rdoc. Avoid CONFLICTS. Bump PKGREVISION.

Revision 1.1 / (download) - annotate - [select for diffs], Fri Jul 14 12:55:45 2017 UTC (3 years ago) by joerg
Branch: MAIN

Collect patches for xapian in a common subdirectory. Put distinfo for
modules into a separate file as well. Don't hard-code -lstdc++ for
broken ancient OpenBSD versions of GCC. Sync p5-Xapian PLIST with
reality.

This form allows you to request diff's between any two revisions of a file. You may select a symbolic revision name using the selection box or you may type in a numeric name using the type-in text box.




CVSweb <webmaster@jp.NetBSD.org>