CVS log for pkgsrc/pkgtools/pbulk/files/pbulk/pbuild/jobs.c

Up to [cvs.NetBSD.org] / pkgsrc / pkgtools / pbulk / files / pbulk / pbuild

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

pbulk-0.57: switch to a binary heap for the build queue

pbulk-base-0.56: Support for adjusting scheduling

Switch to a weighted scheduling algorithm. Before, build order was based
on number of reachable nodes in the dependency graph. This assumes that
heavy packages are required by other packages. Some big packages
nowadays violate that assumption and can result in long periods at the
end of a build where only a few builders are active. Annotating those
packages with PBULK_WEIGHT in the pbulk-index output can boost their
priority to let them be built earlier. The default weight is 100.

Note: the pbulk-build report has grown an extra field per line with the
computed effective weight of each package. This file is normally used
only internally.

pbulk-base-0.55: Optimize DAG computation

Before the build starts, pbulk-build computes the size of the dependee
graph for each package. This is naturally a O(n^2) problem. The existing
algorithm used a linked list to check for duplicates. Replace this with
a simple array for seen markers. While it is still quadratic to reset
the array for every package, clearing the array is a simple memset.
A no-op run after a full build now needs 0.3s on my work station
compared to over 3min before.

Simplify DAG size computation. NFC

pbulk-base-0.50: Don't go into master mode if there is nothing to do.

Format string annotations.

Update pbulk to 0.51 and pbulk-base to 0.46. Add support for reusing old
scan results to speed up the process. Fix a bug that overwrote the
package.log during installation.

pbulk-base-0.44: Fix hash function to actually do proper hashing.
Do some further tweaking to improve matching by storing string size.
While here, add the necessary .WAIT to allow parallel builds.

pbulk-base-0.41:
Improve diagnostic message for dependency cycles by actually showing the
path.

pbulk-base-0.40:
Don't print all failed/successful jobs at the beginning, just a small
message when it is done.

pbulk-0.37:
- replace libevent dependency with internal event loop to make the pbulk
  core standalone
- open success and error log with O_APPEND, which allows e.g. cleaning
  up the error build during build without breaking it if done with care.

pbulk-0.34:
Isolate rules to compute the restricted subset in the upload script.
Make the upload script independent from the report file to allow
uploading partial builds.
Only depend on NO_BIN_ON_FTP to decide what to upload and what not.

Improve verbose mode:
- sprinkle whitespace so pkgnames get neatly aligned
- prepend a progress counter to printfs

In error messages, it is important to be as specific as possible, to
help the user understand the problem.

Added some comments that explain the data structures.

Shorten wording.

Fix grammar. From salo@

Fix a typo. Improve heading for top ten.

Use libnbcompat and fix build issues on Solaris.
Don't depend on sin_len. Slightly factor out code for setting
non-blocking IO.

Bump to pbulk-0.2. Tested by dmcmahill@.

Initial import of pbulk, the new pkgsrc bulk build framework.

Discussion of various parts of the design with jlam@, wiz@, tls@
and many other developers. Special thanks to David Maxwell for
testing the initial prototype and finding some bugs with
Coverity Prevent.

Supported by Google's Summer of Code 2007 project.

OK for import during the freeze: jlam@

From DESCR:
pbulk is the modular bulk build framework for pkgsrc.

This package contains:

- pbulk-scan, a program to scan the entire pkgsrc tree
  or a list of directories therein for packages and
  dependencies. Distributed operation using a master/client
  mode is supported.

- pbulk-resolve, a program to resolve the dependencies
  from the output of pbulk-scan

- pbulk-build, the build scheduler. Distributed builds via TCP
  are supported.

- bulkbuild and related scripts to implement full and limited
  bulk builds on top of those programs.

The pbulk framework is considered experimental.

Initial revision