Up to [cvs.NetBSD.org] / pkgsrc / devel / py-joblib
Request diff between arbitrary revisions
Keyword substitution: kv
Default branch: MAIN
py-joblib: updated to 1.4.2 Release 1.4.2 -- 2024/05/02 Due to maintenance issues, 1.4.1 was not valid and we bumped the version to 1.4.2 - Fix a backward incompatible change in ``MemorizedFunc.call`` which needs to return the metadata. Also make sure that ``NotMemorizedFunc.call`` return an empty dict for metadata for consistency. https://github.com/joblib/joblib/pull/1576
py-joblib: updated to 1.4.0 Release 1.4.0 -- 2024/04/08 - Allow caching co-routines with `Memory.cache`. - Try to cast ``n_jobs`` to int in parallel and raise an error if it fails. This means that ``n_jobs=2.3`` will now result in ``effective_n_jobs=2`` instead of failing. - Ensure that errors in the task generator given to Parallel's call are raised in the results consumming thread. - Adjust codebase to NumPy 2.0 by changing ``np.NaN`` to ``np.nan`` and importing ``byte_bounds`` from ``np.lib.array_utils``. - The parameter ``return_as`` in ``joblib.Parallel`` can now be set to ``generator_unordered``. In this case the results will be returned in the order of task completion rather than the order of submission. - dask backend now supports ``return_as=generator`` and ``return_as=generator_unordered``. - Vendor cloudpickle 3.0.0 and end support for Python 3.7 which has reached end of life.
devel/py-joblib: Unbreak hardware core count Patch in platform-specific cases for NetBSD and FreeBSD Without this, the core count routine errors out No other changes Maintainer feedback timeout (12 days) Issue was discussed on tech-pkg and upstream https://github.com/joblib/joblib/issues/1535
py-joblib: updated to 1.3.2 Release 1.3.2 -- 2023/08/08 Fix a regression in joblib.Parallel introduced in 1.3.0 where explicitly setting n_jobs=None was not interpreted as "unset". Fix a regression in joblib.Parallel introduced in 1.3.0 where joblib.Parallel logging methods exposed from inheritance to joblib.Logger didn't work because of missing logger initialization. Various maintenance updates to the doc, the ci and the test. Release 1.3.1 -- 2023/06/29 Fix compatibility with python 3.7 by vendor loky 3.4.1 which is compatible with this version. Release 1.3.0 -- 2023/06/28 Ensure native byte order for memmap arrays in joblib.load. Add ability to change default Parallel backend in tests by setting the JOBLIB_TESTS_DEFAULT_PARALLEL_BACKEND environment variable. Fix temporary folder creation in joblib.Parallel on Linux subsystems on Windows which do have /dev/shm but don't have the os.statvfs function Drop runtime dependency on distutils. distutils is going away in Python 3.12 and is deprecated from Python 3.10 onwards. This import was kept around to avoid breaking scikit-learn, however it's now been long enough since scikit-learn deployed a fixed (verion 1.1 was released in May 2022) that it should be safe to remove this. A warning is raised when a pickling error occurs during caching operations. In version 1.5, this warning will be turned into an error. For all other errors, a new warning has been introduced: joblib.memory.CacheWarning. Avoid (module, name) collisions when caching nested functions. This fix changes the module name of nested functions, invalidating caches from previous versions of Joblib. Add cache_validation_callback in :meth:`joblib.Memory.cache`, to allow custom cache invalidation based on the metadata of the function call. Add a return_as parameter for Parallel, that enables consuming results asynchronously. Improve the behavior of joblib for n_jobs=1, with simplified tracebacks and more efficient running time. Add the parallel_config context manager to allow for more fine-grained control over the backend configuration. It should be used in place of the parallel_backend context manager. In particular, it has the advantage of not requiring to set a specific backend in the context manager. Add items_limit and age_limit in :meth:`joblib.Memory.reduce_size` to make it easy to limit the number of items and remove items that have not been accessed for a long time in the cache. Deprecate bytes_limit in Memory as this is not automatically enforced, the limit can be directly passed to :meth:`joblib.Memory.reduce_size` which needs to be called to actually enforce the limit. Vendor loky 3.4.0 which includes various fixes. Various updates to the documentation and to benchmarking tools. Move project metadata to pyproject.toml. Add more tests to improve python nogil support.
py-joblib: updated to 1.2.0 Release 1.2.0 Fix a security issue where eval(pre_dispatch) could potentially run arbitrary code. Now only basic numerics are supported. https://github.com/joblib/joblib/pull/1327 Make sure that joblib works even when multiprocessing is not available, for instance with Pyodide https://github.com/joblib/joblib/pull/1256 Avoid unnecessary warnings when workers and main process delete the temporary memmap folder contents concurrently. https://github.com/joblib/joblib/pull/1263 Fix memory alignment bug for pickles containing numpy arrays. This is especially important when loading the pickle with mmap_mode != None as the resulting numpy.memmap object would not be able to correct the misalignment without performing a memory copy. This bug would cause invalid computation and segmentation faults with native code that would directly access the underlying data buffer of a numpy array, for instance C/C++/Cython code compiled with older GCC versions or some old OpenBLAS written in platform specific assembly. https://github.com/joblib/joblib/pull/1254 Vendor cloudpickle 2.2.0 which adds support for PyPy 3.8+. Vendor loky 3.3.0 which fixes several bugs including: robustly forcibly terminating worker processes in case of a crash (https://github.com/joblib/joblib/pull/1269); avoiding leaking worker processes in case of nested loky parallel calls; reliability spawn the correct number of reusable workers.
*: bump PKGREVISION for egg.mk users They now have a tool dependency on py-setuptools instead of a DEPENDS
py-joblib: updated to 1.1.0 Release 1.1.0 Fix byte order inconsistency issue during deserialization using joblib.load in cross-endian environment: the numpy arrays are now always loaded to use the system byte order, independently of the byte order of the system that serialized the pickle. https://github.com/joblib/joblib/pull/1181 Fix joblib.Memory bug with the ignore parameter when the cached function is a decorated function. https://github.com/joblib/joblib/pull/1165 Fix joblib.Memory to properly handle caching for functions defined interactively in a IPython session or in Jupyter notebook cell. https://github.com/joblib/joblib/pull/1214 Update vendored loky (from version 2.9 to 3.0) and cloudpickle (from version 1.6 to 2.0) https://github.com/joblib/joblib/pull/1218
py-joblib: updated to 1.0.1 1.0.1 dask: avoid redundant scattering of large arguments to make a more efficient use of the network resources and avoid crashing dask with “OSError: [Errno 55] No buffer space available” or “ConnectionResetError: [Errno 104] connection reset by peer”.
py-joblib: updated to 1.0.0 1.0.0 Make joblib.hash and joblib.Memory caching system compatible with numpy >= 1.20.0. Also make it explicit in the documentation that users should now expect to have their joblib.Memory cache invalidated when either joblib or a third party library involved in the cached values definition is upgraded. In particular, users updating joblib to a release that includes this fix will see their previous cache invalidated if they contained reference to numpy objects. Remove deprecated check_pickle argument in delayed.
py-joblib: updated to 0.17.0 Release 0.17.0 - Fix a spurious invalidation of `Memory.cache`'d functions called with `Parallel` under Jupyter or IPython. - Bump vendored loky to 2.9.0 and cloudpickle to 1.6.0. In particular this fixes a problem to add compat for Python 3.9.
py-joblib: updated to 0.16.0 Release 0.16.0 -------------- - Fix a problem in the constructors of of Parallel backends classes that inherit from the `AutoBatchingMixin` that prevented the dask backend to properly batch short tasks. - Fix a problem in the way the joblib dask backend batches calls that would badly interact with the dask callable pickling cache and lead to wrong results or errors. - Prevent a dask.distributed bug from surfacing in joblib's dask backend during nested Parallel calls (due to joblib's auto-scattering feature) - Workaround for a race condition after Parallel calls with the dask backend that would cause low level warnings from asyncio coroutines:
py-joblib: updated 0.15.1 Release 0.15.1 - Make joblib work on Python 3 installation that do not ship with the lzma package in their standard library.
py-joblib: updated to 0.15.0 Release 0.15.0 - Drop support for Python 2 and Python 3.5. All objects in ``joblib.my_exceptions`` and ``joblib.format_stack`` are now deprecated and will be removed in joblib 0.16. Note that no deprecation warning will be raised for these objects Python < 3.7. - Fix many bugs related to the temporary files and folder generated when automatically memory mapping large numpy arrays for efficient inter-process communication. In particular, this would cause `PermissionError` exceptions to be raised under Windows and large leaked files in `/dev/shm` under Linux in case of crash. - Make the dask backend collect results as soon as they complete leading to a performance improvement: - Fix the number of jobs reported by ``effective_n_jobs`` when ``n_jobs=None`` called in a parallel backend context. - Upgraded vendored cloupickle to 1.4.1 and loky to 2.8.0. This allows for Parallel calls of dynamically defined functions with type annotations in particular.
py-joblib: updated to 0.14.1 Release 0.14.1 Configure the loky workers' environment to mitigate oversubsription with nested multi-threaded code in the following case: allow for a suitable number of threads for numba (NUMBA_NUM_THREADS); enable Interprocess Communication for scheduler coordination when the nested code uses Threading Building Blocks (TBB) (ENABLE_IPC=1) https://github.com/joblib/joblib/pull/951 Fix a regression where the loky backend was not reusing previously spawned workers. https://github.com/joblib/joblib/pull/968 Revert https://github.com/joblib/joblib/pull/847 to avoid using pkg_resources that introduced a performance regression under Windows: https://github.com/joblib/joblib/issues/965
py-joblib: updated to 0.14.0 Release 0.14.0 Improved the load balancing between workers to avoid stranglers caused by an excessively large batch size when the task duration is varying significantly (because of the combined use of joblib.Parallel and joblib.Memory with a partially warmed cache for instance). https://github.com/joblib/joblib/pull/899 Add official support for Python 3.8: fixed protocol number in Hasher and updated tests. Fix a deadlock when using the dask backend (when scattering large numpy arrays). https://github.com/joblib/joblib/pull/914 Warn users that they should never use joblib.load with files from untrusted sources. Fix security related API change introduced in numpy 1.6.3 that would prevent using joblib with recent numpy versions. https://github.com/joblib/joblib/pull/879 Upgrade to cloudpickle 1.1.1 that add supports for the upcoming Python 3.8 release among other things. https://github.com/joblib/joblib/pull/878 Fix semaphore availability checker to avoid spawning resource trackers on module import. https://github.com/joblib/joblib/pull/893 Fix the oversubscription protection to only protect against nested Parallel calls. This allows joblib to be run in background threads. https://github.com/joblib/joblib/pull/934 Fix ValueError (negative dimensions) when pickling large numpy arrays on Windows. https://github.com/joblib/joblib/pull/920 Upgrade to loky 2.6.0 that add supports for the setting environment variables in child before loading any module. https://github.com/joblib/joblib/pull/940 Fix the oversubscription protection for native libraries using threadpools (OpenBLAS, MKL, Blis and OpenMP runtimes). The maximal number of threads is can now be set in children using the inner_max_num_threads in parallel_backend. It defaults to cpu_count() // n_jobs.
py-joblib: updated to 0.13.2 0.13.2: Add a non-regression test, reporting that cloudpickle versions between 0.5.4 and 0.7 introduced a bug where global variables changes in a parent process between two calls to joblib.Parallel would not be propagated into the workers 0.13.1: Memory now accepts pathlib.Path objects as location parameter. Also, a warning is raised if the returned backend is None while location is not None. Make Parallel raise an informative RuntimeError when the active parallel backend has zero worker. Make the DaskDistributedBackend wait for workers before trying to schedule work. This is useful in particular when the workers are provisionned dynamically but provisionning is not immediate (for instance using Kubernetes, Yarn or an HPC job queue).
py-joblib: updated to 0.13.0 Release 0.13.0: Fix nested backend in SequentialBackend to avoid changing the default backend to Sequential. Fix nested_backend behavior to avoid setting the default number of workers to -1 when the backend is not dask. Release 0.12.5 Include loky 2.3.1 with better error reporting when a worker is abruptly terminated. Also fixes spurious debug output. Include cloudpickle 0.5.6. Fix a bug with the handling of global variables by locally defined functions. Release 0.12.4 Include loky 2.3.0 with many bugfixes, notably w.r.t. when setting non-default multiprocessing contexts. Also include improvement on memory management of long running worker processes and fixed issues when using the loky backend under PyPy. Raises a more explicit exception when a corrupted MemorizedResult is loaded. Loading a corrupted cached file with mmap mode enabled would recompute the results and return them without memmory mapping. Release 0.12.3 Fix joblib import setting the global start_method for multiprocessing. Fix MemorizedResult not picklable. Fix Memory, MemorizedFunc and MemorizedResult round-trip pickling + unpickling. Fixed a regression in Memory when positional arguments are called as kwargs several times with different values. Integration of loky 2.2.2 that fixes issues with the selection of the default start method and improve the reporting when calling functions with arguments that raise an exception when unpickling. Prevent MemorizedFunc.call_and_shelve from loading cached results to RAM when not necessary. Results in big performance improvements
py-joblib: updated to 0.12.2 Release 0.12.2: Integrate loky 2.2.0 to fix regression with unpicklable arguments and functions reported by users. Loky 2.2.0 also provides a protection against memory leaks long running applications when psutil is installed. Joblib now includes the code for the dask backend which has been updated to properly handle nested parallelism and data scattering at the same time. Restored some private API attribute and arguments (MemorizedResult.argument_hash and BatchedCalls.__init__'s pickle_cache) for backward compat. Fix a deprecation warning message (for Memory's cachedir). Release 0.12.1: Make sure that any exception triggered when serializing jobs in the queue will be wrapped as a PicklingError as in past versions of joblib. Fix kwonlydefaults key error in filter_args
devel/py-joblib: Import version 0.12.0 Joblib is a set of tools to provide lightweight pipelining in Python. In particular, joblib offers transparent disk-caching of the output values and lazy re-evaluation (memoize pattern), easy simple parallel computing, and logging and tracing of the execution. Joblib is optimized to be fast and robust in particular on large data and has specific optimizations for numpy arrays. Packaged by Kamel Ibn Aziz Derouiche for pkgsrc-wip and updated by me.