CVS log for src/sys/dev/raidframe/rf_raid.h

Up to [cvs.NetBSD.org] / src / sys / dev / raidframe

Request diff between arbitrary revisions

Keyword substitution: kv
Default branch: MAIN

Pull up following revision(s) (requested by oster in ticket #674):

	sys/dev/raidframe/rf_raid.h: revision 1.52
	sbin/raidctl/raidctl.8: revision 1.80
	sys/dev/raidframe/rf_driver.c: revision 1.141
	sys/dev/raidframe/rf_disks.c: revision 1.94
	sys/dev/raidframe/rf_diskqueue.c: revision 1.64
	sys/dev/raidframe/rf_diskqueue.h: revision 1.30
	sys/dev/raidframe/rf_disks.h: revision 1.15
	sys/dev/raidframe/rf_netbsdkintf.c: revision 1.414
	sys/dev/raidframe/rf_reconstruct.c: revision 1.129
	sys/dev/raidframe/raidframeio.h: revision 1.12
	sbin/raidctl/raidctl.c: revision 1.79

Implement hot removal of spares and components. From manu@.

Implement a long desired feature of automatically incorporating
a used spare into the array after a reconstruct.

Given the configuration:

Components:
 /dev/wd0e: failed
 /dev/wd1e: optimal
 /dev/wd2e: optimal
Spares:
 /dev/wd3e: spare

Running 'raidctl -F /dev/wd0e raid0' will now result in the
following configuration after a successful rebuild:

Components:
 /dev/wd3e: optimal
 /dev/wd1e: optimal
 /dev/wd2e: optimal
No spares.

Thanks to manu@ for the development of the initial set of changes
which allowed the changes to automatically incorporate a used spare
to come to fruition. Thanks also to manu@ for useful discussions
about and additional testing of these changes.

We no longer need the deprecated copyback functionality now that
incorporating a used spare is automatic.

Copyback has always been an issue, as to do a copyback all IO to
the array had to be suspended, and so was very, very unlikely to
have been used in anything resembling a production system.

Implement hot removal of spares and components.  From manu@.

Implement a long desired feature of automatically incorporating
a used spare into the array after a reconstruct.

Given the configuration:
Components:
           /dev/wd0e: failed
           /dev/wd1e: optimal
           /dev/wd2e: optimal
Spares:
           /dev/wd3e: spare

Running 'raidctl -F /dev/wd0e raid0' will now result in the
following configuration after a successful rebuild:
Components:
           /dev/wd3e: optimal
           /dev/wd1e: optimal
           /dev/wd2e: optimal
No spares.

Thanks to manu@ for the development of the initial set of changes
which allowed the changes to automatically incorporate a used spare
to come to fruition.  Thanks also to manu@ for useful discussions
about and additional testing of these changes.

Merge thorpej-cfargs2.

Sync w/ HEAD.

Accidentally commited some other changes that weren't quite ready.  Add
these changes to fix the build.

Sync with HEAD.

Extensive mechanical changes to the pools used in RAIDframe.

Alloclist remains not per-RAID, so initialize that pool
separately/differently than the rest.

The remainder of pools in RF_Pools_s are now per-RAID pools.  Mostly
mechanical changes to functions to allocate/destroy per-RAID pools.
Needed to make raidPtr available in certain cases to be able to find
the per-RAID pools.

Extend rf_pool_init() to now populate a per-RAID wchan value that is
unique to each pool for a given RAID device.

TODO: Complete the analysis of the minimum number of items that are
required for each pool to allow IO to progress (i.e. so that a request
for pool resources can always be satisfied), and dynamically scale
minimum pool sizes based on RAID configuration.

Mostly merge changes from HEAD upto 20200411

fix the function pointer and callback mess:
- callback functions return 0 and their result is not checked; make them void.
- there are two types of callbacks and they used to overload their parameters
  and the callback structure; separate them into "function" and "value"
  callbacks.
- make the wait function signature consistent.

Sync with HEAD

Shuffle softc declarations to a different .h file.  Create missing
rf_get_raid().  Things compile, but don't work correctly.

Synch with HEAD

remove the final tsleep/wakeup pair in raidframe.

update from HEAD

src is too big these days to tolerate superfluous apostrophes.  It's
"its", people!

sync with head.

for a reference, the tree before this commit was tagged
as yamt-pagecache-tag8.

this commit was splitted into small chunks to avoid
a limitation of cvs.  ("Protocol error: too many arguments")

resync from head

allocate devices dynamically.

Sync with HEAD.

sync with head

convert the main raidPtr mutex to a kmutex, and add a couple of cv's to
cover the old sleep/wakeup points for adding_hot_spare and waitForReconCond.
convert all remaining simple_lock's to kmutexes (they're not used or compiled
right now... even with all options enabled) and remove the support for them.

this leaves just a pair of tsleep()/wakeup() calls using old scheduling APIs.

convert access_suspend_mutex to a kmutex/cv.

- convert rf_printf_mutex to a kmutex
- convert rf_rad_lock and the per-raid "cv" to per-raid kmutex/and real cv
- use rf_mutex_init() in places, and move it with the similar definitions

prepare to convert more raidframe old lock/sleep APIs to mutex/condvar:

- remove RF_DECLARE_EXTERN_MUTEX and RF_DECLARE_STATIC_MUTEX, the qualifier
  can be provided at the use point with the normal define
- rename the *LGMGR_MUTEX() macros to *mutex2() names, and add some more
  defines for use:
	rf_declare_mutex2()
	rf_declare_cond2()
	rf_lock_mutex2()
	rf_unlock_mutex2()
	rf_init_mutex2()
	rf_destroy_mutex2()
	rf_init_cond2()
	rf_destroy_cond2()
	rf_wait_cond2()
	rf_signal_cond2()
	rf_broadcast_cond2()
- use the new names for the configureMutex(), which previous used some combo
  of direct mutex* calls and macros
- convert the node_queue to use a mutex/cv combo
- in rf_ShutdownEngine() and DAGExecutionThread(), also signal the former from
  the latter when it is done and about to exit
- convert iodone_lock to use the new macros

convert the iodone_lock to a mutex, and use a condvar for signalling.

this only handles the smallest use of old simple_lock/tsleep/wakeup
APIs inside raidframe, and it points out that cv(9)'s have only one
wait channel per cv, whereas each tsleep() caller can specify a
different wait channel.  this change removes the difference between
normal raidio and waiting for IO during shutdown.

i've tested this one 3 systems, ran atf, and had mlelstv and rmind
review the change.

sync to netbsd-5

sync with head

Pull up following revision(s) (requested by tron in ticket #1187):
	sbin/raidctl/raidctl.8: revisions 1.57-1.59 via patch
	sbin/raidctl/raidctl.c: revision 1.42 via patch
	sys/dev/raidframe/files.raidframe: revision 1.8 via patch
	sys/dev/raidframe/rf_copyback.c: revision 1.42 via patch
	sys/dev/raidframe/rf_disks.c: revision 1.72 via patch
	sys/dev/raidframe/rf_driver.c: revision 1.122 via patch
	sys/dev/raidframe/rf_engine.c: revision 1.40 via patch
	sys/dev/raidframe/rf_kintf.h: revision 1.21 via patch
	sys/dev/raidframe/rf_netbsdkintf.c: revision 1.269 via patch
	sys/dev/raidframe/rf_paritymap.c: revisions 1.1-1.3 via patch
	sys/dev/raidframe/rf_paritymap.h: revision 1.1 via patch
	sys/dev/raidframe/rf_parityscan.c: revision 1.33 via patch
	sys/dev/raidframe/rf_parityscan.h: revision 1.8 via patch
	sys/dev/raidframe/rf_raid.h: revision 1.38 via patch
	sys/dev/raidframe/rf_reconstruct.c: revision 1.108 via patch
	sys/dev/raidframe/rf_states.c: revision 1.44 via patch
	sys/dev/raidframe/raidframeio.h: revision 1.6 via patch
	sys/dev/raidframe/raidframevar.h: revision 1.13 via patch
Pull up the RAIDframe parity map Summer Of Code project.
Drastically reduces the amount of time spent rewriting parity after an
unclean shutdown by keeping better track of which regions might have had
outstanding writes.  Enabled by default; can be disabled on a per-set
basis, or tuned, with the new raidctl(8) commands.

Finally commit the RAIDframe parity map Summer Of Code project.

Drastically reduces the amount of time spent rewriting parity after an
unclean shutdown by keeping better track of which regions might have had
outstanding writes.  Enabled by default; can be disabled on a per-set
basis, or tuned, with the new raidctl(8) commands.

Discussed on tech-kern@ to a general air of approval; exhortations to
commit from mrg@, christos@, and others.

Thanks to Google for their sponsorship, oster@ for mentoring the
project, assorted developers for trying very hard to break it, and
probably more I'm forgetting.

sync with HEAD

sync with head.

Sync with head.

Sync with HEAD.

Make a couple of variables 64-bit quantities to avoid overflow issues
when arrays have a large number of stripes.  Remove unused variable
(recon_stripes_done).  Thanks to Dieter Roelants for the report,
suggestions, and testing.  Addresses PR#36878.

sync with head.

Sync with HEAD.

Kill caddr_t; there will be some MI fallout, but it will be fixed shortly.

merge ktrace-lwp.

sync with -current

sync with head.  xen and whitespace.  xen part is not finished.

Sync with HEAD.

Hi Perry!

nuke trailing whitespace

Fix the sync with head I botched.

Sync with HEAD.

Sync with HEAD

Pull up revision 1.33 (requested by oster in ticket #123):
These changes complete the effective removal of malloc() from all
write paths within RAIDframe.  They also resolve the "panics with
RAID 5 sets with more than 3 components" issue which was present
(briefly) in the commits which were previously supposed to address
the malloc() issue.
With this new code the 5-component RAID 5 set panics are now gone.
It is also now also possible to swap to RAID 5.
The changes made are:
1) Introduce rf_AllocStripeBuffer() and rf_FreeStripeBuffer() to
allocate/free one stripe's worth of space.  rf_AllocStripeBuffer() is
used in rf_MapUnaccessedPortionOfStripe() where it is not sufficient to
allocate memory using just rf_AllocBuffer().  rf_FreeStripeBuffer() is
called from rf_FreeRaidAccDesc(), well after the DAG is finished.
2) Add a set of emergency "stripe buffers" to struct RF_Raid_s.
Arrange for their initialization in rf_Configure().  In low-memory
situations these buffers will be returned by rf_AllocStripeBuffer()
and re-populated by rf_FreeStripeBuffer().
3) Move	RF_VoidPointerListElem_t *iobufs from the dagHeader into
into struct RF_RaidAccessDesc_s.  This is more consistent with the
original code, and will not result in items being freed "too early".
4) Add a RF_RaidAccessDesc_t *desc to RF_DagHeader_s so that we have a
way to find desc->iobufs.
5) Arrange for desc in the DagHeader to be initialized in InitHdrNode().
6) Don't cleanup iobufs in rf_FreeDAG() -- the freeing is now delayed
until rf_FreeRaidAccDesc() (which is how the original code handled the
allocList, and for which there seem to be some subtle, undocumented
assumptions).
7) Rename rf_AllocBuffer2() to be rf_AllocBuffer() and remove the
former rf_AllocBuffer().  Fix all callers of rf_AllocBuffer().
(This was how it was *supposed* to be after the last time these
changes were made, before they were backed out).
8) Remove RF_IOBufHeader and all references to it.
9) Remove desc->cleanupList and all references to it.
Fixes PR#20191

These changes complete the effective removal of malloc() from all
write paths within RAIDframe.  They also resolve the "panics with
RAID 5 sets with more than 3 components" issue which was present
(briefly) in the commits which were previously supposed to address
the malloc() issue.

With this new code the 5-component RAID 5 set panics are now gone.

It is also now also possible to swap to RAID 5.

The changes made are:

1) Introduce rf_AllocStripeBuffer() and rf_FreeStripeBuffer() to
allocate/free one stripe's worth of space.  rf_AllocStripeBuffer() is
used in rf_MapUnaccessedPortionOfStripe() where it is not sufficient to
allocate memory using just rf_AllocBuffer().  rf_FreeStripeBuffer() is
called from rf_FreeRaidAccDesc(), well after the DAG is finished.

2) Add a set of emergency "stripe buffers" to struct RF_Raid_s.
Arrange for their initialization in rf_Configure().  In low-memory
situations these buffers will be returned by rf_AllocStripeBuffer()
and re-populated by rf_FreeStripeBuffer().

3) Move	RF_VoidPointerListElem_t *iobufs from the dagHeader into
into struct RF_RaidAccessDesc_s.  This is more consistent with the
original code, and will not result in items being freed "too early".

4) Add a RF_RaidAccessDesc_t *desc to RF_DagHeader_s so that we have a
way to find desc->iobufs.

5) Arrange for desc in the DagHeader to be initialized in InitHdrNode().

6) Don't cleanup iobufs in rf_FreeDAG() -- the freeing is now delayed
until rf_FreeRaidAccDesc() (which is how the original code handled the
allocList, and for which there seem to be some subtle, undocumented
assumptions).

7) Rename rf_AllocBuffer2() to be rf_AllocBuffer() and remove the
former rf_AllocBuffer().  Fix all callers of rf_AllocBuffer().
(This was how it was *supposed* to be after the last time these
changes were made, before they were backed out).

8) Remove RF_IOBufHeader and all references to it.

9) Remove desc->cleanupList and all references to it.

Fixes PR#20191

For each RAID set, pre-allocate a number of "emergency buffers" to be
used in the event that we can't malloc a buffer of the appropriate
size in the traditional way.  rf_AllocIOBuffer() and rf_FreeIOBuffer()
deal with allocating/freeing these structures.  These buffers are
stored in a list on the 'iobuf' list.  iobuf_count keeps track of how
many buffers are available, and numEmergencyBuffers is the effective
"high-water" mark for the freelist.  The buffers allocated by
rf_AllocIOBuffer() are stripe-unit sized, which is the maximum
size requested by any of the callers.

Add an iobufs entry to RF_DagHeader_s.  Use it for keeping track of
buffers that get allocated from the free-list.

Add a "generic list" pool (VoidPointerListElement Pool) for elements
used to maintain a list of allocated memory.  [It is somewhat less
than ideal to add another little pool to handle this...]

Teach rf_AllocBuffer() to use the new rf_AllocIOBuffer().  Modify
other Mallocs to use rf_AllocIOBuffer(), and to update dag_h->iobufs as
appropriate.

Update rf_FreeDAG() to handle cleanup of dag_h->iobufs.

While here, add some missing pool_destroy() calls for a number of pools.

With these changes, it should (in theory) be possible to swap on
RAID 5 sets again.  That said, I've not had any success there yet --
but the last issue I saw at least wasn't in RAIDframe. :-}

[There is room for this code to become a bit more consise, but I
wanted to do a checkpoint here with something known to work :) ]

dag_node_pool never did get used here.  Turf.

 - don't use rf_PrintUserStats() for recon statistics.
rf_PrintUserStats() was mean for the simulator, and doesn't provide
any real info in kernel-space, especially for reconstructs.
Reconstructing actually renders the stats even more useless, since it
resets them all to zero before the reconstruct starts!

 - since rf_PrintUserStats() is no longer used, nuke it along with the
routines that feed it.  Nothing was using this code, and if we ever
need it again, we know where to find it.

Move pss_pool to rf_pools.  Will save a bit of extra memory at
run-time, and we can only do one reconstruction at a time anyway.
Nuke pss_issued_pool - move it to an internal structure in pss.

A few more cleanups missed in last commit.

Minor cleanup.  No functional change.

Use RF_ACC_TRACE to #if out more chunks of code related only
to access tracing.  (not turned on yet)

iCleanup the RF_CREATE_PARAM3().  Middle two "arguments" were nothing
but 0 in all cases.

remove terminate_disk_queues from RF_Raid_s.  The hist_diskreq[]
stuff is only used for the CHAINDECLUSTERING, so hide it with an #if.

Nuke a bunch of unused variables:
 - node_queue_cond
 - quiescent_cond
 - eq_cond
 - desc->cond
 - desc->head
 - diskqueue->numWaiting

Nuke rf_print_unable_to_init_cond().
Nuke rf_TerminateDiskQueues prototype from rf_diskqueue.h.

sparediskids isn't used anywhere. turf.

Garbage-collect a whole mess of this RF_THREADGROUP_* stuff that isn't
being used.  Then, nuke rf_init_managed_threadgroup() and all
descendants and relations.

Let's see...  raidPtr->recon_done_procs is never set to anything
(other than NULL when raidPtr is initialized).  That means
SignalReconDone() never does anything useful.  Bye-bye!

Say good-bye to recon_done_procs and recon_done_procs_mutex (and its
initializer) as well.

[Having received a definite lack of strenuous objection, a small amount
of strenuous agreement, and some general agreement, this commit is
going ahead because it's now starting to block some other changes I
wish to make.]

Remove most of the support for the concept of "rows" from RAIDframe.
While the "row" interface has been exported to the world, RAIDframe
internals have really only supported a single row, even though they
have feigned support of multiple rows.

Nothing changes in configuration land -- config files still need to
specify a single row, etc.  All auto-config structures remain fully
forward/backwards compatible.

The only visible difference to the average user should be a
reduction in the size of a GENERIC kernel (i386) by 4.5K.  For those
of us trolling through RAIDframe kernel code, a lot of the driver
configuration code has become a LOT easier to read.

constify some

Catch up to -current

Better protect hot-spare adding, and make it LOCKDEBUG friendly.

Catch up to -current.

poolify the allocation of Parity Stripe Status structures.

XXX: Current code may have problems if kernel memory is completely depleted.
This is, unfortunately, not the only chunk of RAIDframe code to have
this problem, and will have to be dealt with. :(

As part of the effort to get RAIDframe playing nicely with LOCKDEBUG,
rework how completed requests are handled.  In particular, instead of
doing all sorts of work and locking in interrupt context, completed
requests are now queued.  A new kernel thread (rf_RaidIOThread) now
handles calling rf_DiskIOComplete() and (req->CompleteFunc)() for each
completed request.  There is still work to be done to make RAIDframe
LOCKDEBUG friendly, but this change is a huge step forward.

Reviewed by (and many thanks to): thorpej

Sync kqueue branch with -current.

Catch up to -current.

Catch up with -current. Fix some bogons in the sparc64 kbd/ms
attach code. cd18xx conversion provided by mrg.

Step 2 of the disentanglement.  We now look to <dev/raidframe/*> for
the stuff that used to live in rf_types.h, rf_raidframe.h, rf_layout.h,
rf_netbsd.h, rf_raid.h, rf_decluster,h, and a few other places.
Believe it or not, when this is all done, things will be cleaner.

No functional changes to RAIDframe.

Catch up with -current.

Catch up to -current.
Again.

RF_DEV2RAIDID isn't used anywhere anymore.  Bye bye.

Update thorpej_scsipi to -current as of a month ago
A i386 GENERIC kernel compiles without the siop, ahc and bha drivers
(will be updated later). i386 IDE/ATAPI and ncr work, as well as
sparc/esp_sbus. alpha should work as well (untested yet).
siop, ahc and bha will be updated once I've updated the branch to current
-current, as well as machine-dependant code.

Oops...  Missed commmitting this last night.  Thanks to Matthias Scheler
for letting me know.
(Add the DiskQueueSW pointer)

Be more aggressive about updating component labels in the event
of a real component failure (or a simulated failure):
- add 'numNewFailures' to keep track of the number of disk failures
since mod_counter was last updated for each component label.
- make sure we call rf_update_component_labels() upon any component failure,
real or simulated.

- cleanup of more component label code
- make current default label values available everywhere
- make sure numBlocks and blockSize in component labels get initialized
for all component labels
- check for component size to be smaller than or equal to the partition size
when autoconfiguring

Get recent changes into the tree:
- make component_label variables more consistent (==> clabel)
- re-work incorrect component configuration code
- re-work disk configuration code
- cleanup initial configuration of raidPtr info
- add auto-detection of components and RAID sets (Disabled, for now)
- allow / on RAID sets (Disabled, for now)
- rename "config_disk_queue" to "rf_ConfigureDiskQueue" and properly prototype
in rf_diskqueue.h
- protect some headers with #if _KERNEL  (XXX this needs to be fixed properly)
  and cleanup header formatting.
- expand the component labels (yes, they should be backward/forward compatible)
- other bits and pieces (some function names are still bogus, and will get
changed soon)

- update RF_CREATE_THREAD to handle a 'process name' argument.
- fire up a new thread for parity re-writes, copybacks, and reconstructs.
  The ioctl's which trigger these actions now return immediately.
- add progress accounting for the above actions.
- minor rototillage of rf_netbsdkintf.c to deal with all of the above.

pull up rev 1.6 from trunk (requested by oster):
  Add a more reasonable throttling mechanism to the RAIDframe code.
  Increases write performance, and helps prevent the I/O routines from
  using too much kernel memory.

Remove a 'struct proc *'-passing abomination that's been bugging me
for quite some time.

Update from trunk.

Once upon a time, long long ago, there was a "fix" added to the
RAIDframe driver to stop it from eating too much kernel memory when
writing data.  But that fix had a nasty side-affect of hurting write
performance (*much* more than I thought it would).  These changes nuke
that "fix", and instead put in a more reasonable mechanism for limiting
the number of simultaneous IO's which can be happening for each RAID device.
The result is a noticeable improvement in write throughput.  The End.

Update for recent changes including component label support, clean
bits, rebuilding components in-place, adding hot spares, shutdownhooks, etc.

Cleanup/remove unused cruft.  First kick at component labels and clean bits.
Still work in progress.  New code is there, but not enabled yet.

Phase 2 of the RAIDframe cleanup.  The source is now closer to KNF
and is much easier to read.  No functionality changes.

RAIDframe cleanup, phase 1.  Nuke simulator support, user-land driver,
out-dated comments, and other unneeded stuff.  This helps prepare
for cleaning up the rest of the code, and adding new functionality.

No functional changes to the kernel code in this commit.

RAIDframe, version 1.1, from the Parallel Data Laboratory at
Carnegie Mellon University.  Full RAID implementation, including
levels 0, 1, 4, 5, 6, parity logging, and a few other goodies.
Ported to NetBSD by Greg Oster.