From owner-newconfig@jp.freebsd.org  Mon May 10 04:43:12 1999
Received: (from daemon@localhost)
	by castle.jp.freebsd.org (8.9.3+3.2W/8.7.3) id EAA21431;
	Mon, 10 May 1999 04:43:12 +0900 (JST)
	(envelope-from owner-newconfig@jp.FreeBSD.org)
Received: from sraigw.sra.co.jp (sraigw.sra.co.jp [202.32.10.2])
	by castle.jp.freebsd.org (8.9.3+3.2W/8.7.3) with ESMTP id EAA21426
	for <newconfig@jp.freebsd.org>; Mon, 10 May 1999 04:43:10 +0900 (JST)
	(envelope-from soda@sra.co.jp)
Received: from srasvf.sra.co.jp (srasvf [133.137.28.2])
	by sraigw.sra.co.jp (8.8.7/3.6Wbeta7-sraigw) with ESMTP id EAA28261;
	Mon, 10 May 1999 04:42:58 +0900 (JST)
Received: from srapc342.sra.co.jp (srapc342 [133.137.28.111])
	by srasvf.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id EAA09441;
	Mon, 10 May 1999 04:42:42 +0900 (JST)
Received: from srapc342 (localhost [127.0.0.1]) by srapc342.sra.co.jp (8.8.8/3.4W-sra) with ESMTP id EAA23320; Mon, 10 May 1999 04:42:53 +0900 (JST)
Message-Id: <199905091942.EAA23320@srapc342.sra.co.jp>
From: Noriyuki Soda <soda@sra.co.jp>
To: newconfig@jp.freebsd.org
Cc: Peter Wemm <peter@netplex.com.au>
Date: Mon, 10 May 1999 04:42:52 +0900
Reply-To: newconfig@jp.freebsd.org
Precedence: list
X-Distribute: distribute version 2.1 (Alpha) patchlevel 24e+990430
X-Sequence: newconfig 44
Subject: [newconfig 44] new-bus/newconfig discussion digest
Errors-To: owner-newconfig@jp.freebsd.org
Sender: owner-newconfig@jp.freebsd.org
X-Originator: soda@sra.co.jp

This is discussion which was brought up between Mr. Peter Wemm and me
before new-bus was official imported to FreeBSD repository.
I'd like to thank Peter for permitting me to forward whole discussion
to this mailing-list.

Please note that on these mails, Peter was discussing as a developer,
not as an official core discussion.  He wrote with the understanding
that it was between newconfig group and Peter.

------------------------------------------------------------

Return-Path: soda@sra.co.jp
Received: from sramhc.sra.co.jp (root@sramhc [133.137.20.31]) by srapc342.sra.co.jp (8.8.8/3.4W-sra) with ESMTP id XAA16919 for <soda@srapc342.sra.co.jp>; Thu, 15 Apr 1999 23:49:32 +0900 (JST)
Received: from srasvf.sra.co.jp (root@srasvf.sra.co.jp [133.137.28.2])
	by sramhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id XAA22410;
	Thu, 15 Apr 1999 23:49:21 +0900 (JST)
Received: from srapc342.sra.co.jp (srapc342 [133.137.28.111])
	by srasvf.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id XAA10098;
	Thu, 15 Apr 1999 23:48:54 +0859 (JST)
Received: (from soda@localhost) by srapc342.sra.co.jp (8.8.8/3.4W-sra) id XAA16913; Thu, 15 Apr 1999 23:49:02 +0900 (JST)
Date: Thu, 15 Apr 1999 23:49:02 +0900 (JST)
Message-Id: <199904151449.XAA16913@srapc342.sra.co.jp>
From: Noriyuki Soda <soda@sra.co.jp>
To: Peter Wemm <peter@netplex.com.au>
CC: Warner Losh <imp@harmony.village.org>, Atsushi Furuta <furuta@sra.co.jp>,
        Noriyuki Soda <soda@sra.co.jp>, Satoshi Asami <asami@freebsd.org>
Subject: new-bus/newconfig
Mime-Version: 1.0 (generated by tm-edit 7.47)
Content-Type: text/plain; charset=US-ASCII

Hi, Peter.

I have some questions about new-bus/newconfig. Could you answer my
questions?
Note that English is not my native language, so please forgive me
about English syntax errors and ambiguous or misleading expressions.


First of all, I think fundamental design difference between new-bus
and newconfig is:

[A] new-bus puts the informations about inter-module dependency and
  bus hierarchy to the kernel module itself.
[B] newconfig puts these informations to another place. i.e. the
  "files" file.

(Note that I didn't say a kernel configuration file, but the "files" file.)

You once wrote the following sentences on freebsd-current mailing-list.

>> There was also the problem of duplication of information.  Drivers
>> in the kernel were written to attach to specific busses and the
>> exact same information had to be duplicated in the user's config
>> file.  Some alternatives (the original config.new, for example)
>> were worse because they added another copy - various Makefiles and
>> rule files needed another set of information to say the same thing
>> again.

But I think this your description is not accurate. The reason why the
newconfig duplicates these informations is that these informations is
needed on both dynamic linking and static linking.
Because the new-bus puts these informations into kernel modules
itself, these informations cannot be used on static linking. 
The newconfig will work well with both dynamic linking and static
linking. But new-bus works well with dynamic linking only.

So, my first question is:

[Question 1]
	Is there any consensus about the following issue in "core" ?
		The fundamental difference about new-bus and newconfig
		is that the new-bus only supports dynamic linking, and
		the newconfig supports both static linking and dynamic 
		linking.
		We are going to new-bus, and will completely avoid
		static linking. (static linking of device drivers will
		not be able to work on FreeBSD, eventually).
	If there are no consensus like this until now, this issue
	shold be approved by "core", before deciding either the new-bus
	or the newconfig.
	The newconfig people think that it is better to support both
	dynamic linking and static linking.
	For example, if one doesn't statically link a console device
	to a kernel, it is difficult to find the reason when dynamic
	linking of the console device is failed.  And also, as a
	matter of fact, completely dynamic configuration is not
	practical for now.
	Will really FreeBSD only support dynamic linking ?
	I've heard that dynamic linking is a only way to go, from
	the newbus people, several times. But do "core" members think
	so, too ?
	As you see the current FreeBSD kernel which is almost
	statically linked, the static linked kernel itself is not so
	bad. And it works better than fully dynamic linked kernel,
	in same cases.

The next issue is how to implement dependency on the new-bus.

Once, Doug wrote the following message on freebsd-mobile.

dfr> From: Doug Rabson <dfr@nlsystems.com>
dfr> Subject: Re: Any success with CirrusLogic 6729/6730???
dfr> Date: Mon, 12 Apr 1999 19:20:54 +0100 (BST)
dfr> Message-ID: <Pine.BSF.4.05.9904121917570.74823-100000@herring.nlsystems.com>
dfr> Delivered-To: freebsd-mobile@freebsd.org
dfr> 
dfr> A split driver can be dynamically loaded by using the dependancy mechanism
dfr> on the KLD modules. Imagine a device foo which support pci and isa
dfr> attachments. One could build three modules:
dfr>        foo.ko          containing the core driver functionality
dfr>        foo_isa.ko      probe/attach for isa bus
dfr>        foo_pci.ko      probe/attach for pci bus
dfr> 
dfr> The two bus-specific modules would depend on the foo.ko module ensuring
dfr> that it is loaded automatically if either bus-specific module is used.

How does the new-bus implement this ?
I have two ideas.

[a1] Put the dependency information to the kernel object file by using
  DT_NEEDED tag in elf object.

[a2] Make the foo_{isa,pci}.ko load the foo.ko.

I think the [a1] is not consistent with the new-bus policy [A].
If the new-bus choose [a1], the dependency information should be
placed on the out side of the kernel C source to produce linker
options which put the dependency information to the kernel object.
If some of the these informations should be duplicated to the
out side of the kernel C source, what's wrong about completely
duplicates these informations to out side of the kernel C source
(as the newconfig)? It has real benefits, these information can 
be used on static linking, and also can be used by other utilities
which cannot be made on the new-bus.
I think the new-bus policy [A] is overkill (which makes static
linking impossible without real reasons) in this case.

If [a2] is the way which the new-bus choose, the functions in the
foo_isa.ko (or foo_pci.ko) cannot directly call the functions in
foo.ko, because the foo.ko might not be loaded when the foo_isa.ko (or
foo_pci.ko) is loaded, then the symbols in the foo.ko which is
refered by the foo_isa.ko will be unresolved. So, in this case, 
the new-bus have to use the indirect call like the DEVMETHOD() 
to call the functions in foo.ko from the functions in foo_isa.ko
(or foo_pci.ko).
I think this is not the better way to implement the kernel. 
In this case, the DEVMETHOD() doesn't do anything really meaningful.
Because both foo_isa.ko and foo.ko shares many data structures, the
DEVMETHOD() way doesn't guarantee the consitency between foo_isa.ko
and foo.ko. And there is other better way to guarantee this.
Note that in this case, the new-bus cannot implement the newconfig shim.
Because the newconfig don't have to use DEVMETHOD() for this.

The newconfig doesn't have these problems. It is consistent because
always handles the dependency information in same way, and it can use
either direct call or indirect call like the DEVMETHOD(), according to
programmer's choise.

So, second questions is

[Question 2]
	Which is the way the new-bus choose? [a1]? [a2]?
	In both cases, the newconfig seems to be better. Isn't it?

And this is the last question.

[Question 3]
	Do the new-bus have a plan how the old-config will be?
	Definitely it is most ugly part of the new-bus.

I cannot read your reply for a while, because it is 23:40pm in Japan,
and I'll go to bed.

Thanks.

P.S.
I think the KLD is the great and the best part of the new-bus.
The newconfig follows the new-bus in this part.

P.P.S.
Warner, forgive me about suddenly CC'ed this mail. :-)

P.P.P.S.
Furuta, if I miss something major issue, please say so.
- --
soda

------------------------------

Return-Path: peter@netplex.com.au
Received: from sramhc.sra.co.jp (root@sramhc [133.137.20.31]) by srapc342.sra.co.jp (8.8.8/3.4W-sra) with ESMTP id DAA19699 for <soda@srapc342.sra.co.jp>; Fri, 16 Apr 1999 03:33:14 +0900 (JST)
Received: from sranha.sra.co.jp (sranha.sra.co.jp [133.137.8.8])
	by sramhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id DAA24420;
	Fri, 16 Apr 1999 03:33:11 +0900 (JST)
Received: from sraigw.sra.co.jp (sraigw-hub [133.137.8.14])
	by sranha.sra.co.jp (8.8.7/3.6Wbeta7-sranha) with ESMTP id DAA28338;
	Fri, 16 Apr 1999 03:33:06 +0900 (JST)
Received: from spinner.netplex.com.au (spinner.netplex.com.au [202.12.86.3])
	by sraigw.sra.co.jp (8.8.7/3.6Wbeta7-sraigw) with ESMTP id DAA23586;
	Fri, 16 Apr 1999 03:32:59 +0900 (JST)
Received: from netplex.com.au (localhost [127.0.0.1])
	by spinner.netplex.com.au (Postfix) with ESMTP
	id 7A7081F61; Fri, 16 Apr 1999 02:32:50 +0800 (WST)
	(envelope-from peter@netplex.com.au)
X-Mailer: exmh version 2.0.2 2/24/98
To: Noriyuki Soda <soda@sra.co.jp>
Cc: Warner Losh <imp@harmony.village.org>, Atsushi Furuta <furuta@sra.co.jp>,
        Satoshi Asami <asami@freebsd.org>
Subject: Re: new-bus/newconfig 
In-reply-to: Your message of "Thu, 15 Apr 1999 23:49:02 +0900."
             <199904151449.XAA16913@srapc342.sra.co.jp> 
Date: Fri, 16 Apr 1999 02:32:50 +0800
From: Peter Wemm <peter@netplex.com.au>
Message-Id: <19990415183252.7A7081F61@spinner.netplex.com.au>

Noriyuki Soda wrote:
> Hi, Peter.
> 
> I have some questions about new-bus/newconfig. Could you answer my
> questions?
> Note that English is not my native language, so please forgive me
> about English syntax errors and ambiguous or misleading expressions.

I appreciate the effort you are making.  I come from a single-language
country so I've been lucky enough to not have needed to deal with language
difficulties.

I apologise in advance for the length of this reply..

> First of all, I think fundamental design difference between new-bus
> and newconfig is:
> 
> [A] new-bus puts the informations about inter-module dependency and
>   bus hierarchy to the kernel module itself.
> [B] newconfig puts these informations to another place. i.e. the
>   "files" file.
> 
> (Note that I didn't say a kernel configuration file, but the "files" file.)

Yes, that would be a fair statement, but simplifies the situation a lot.

A couple of key implications of these design choices as I see it are:

- - newconfig takes those "files" files and generates tables which are
  statically compiled into the kernel.  The possible relationships
  are frozen at that point I believe.
- - The new-bus relationships are dynamically extendable.  You can load
  drivers and busses with relationships that were not even heard of when
  the kernel was compiled.

As an example, suppose you built a newconfig kernel with the current
standard set of usb and pccard driver relationships from the "files.*"
files and you are running it happily on a system that you cannot shut down.
Then, suppose somebody supplies you with a new device which happens to be a
pccard adapter that runs on USB.  ie: a USB->PCCARD bridge that you plug
into a USB port and has two pccard slots.  It is my understanding that in
order to load a driver for this new bridge, you would be stuck since the
configuration engine has been statically configured with 'card*' attaching at
pcic/pci/isa/etc and not the usb bridge you want to load.

An example of this is the PCI->cardbus bridge (cbb) which is defined in
i386/conf/files.i386.newconf and is listed as providing cardslot and
pcmciabus if I read it right, and that this is compiled into the
permanently into kernel.  As I understand it, if this relation wasn't
compiled into the kernel, then it wouldn't be possible to kldload a cbb
driver and have it automatically provide cardbus and pcmcia services.

That is what I believe is the the most critical consequence of the
differences between new-bus and newconfig designs.

> You once wrote the following sentences on freebsd-current mailing-list.
> 
> >> There was also the problem of duplication of information.  Drivers
> >> in the kernel were written to attach to specific busses and the
> >> exact same information had to be duplicated in the user's config
> >> file.  Some alternatives (the original config.new, for example)
> >> were worse because they added another copy - various Makefiles and
> >> rule files needed another set of information to say the same thing
> >> again.
> 
> But I think this your description is not accurate. The reason why the
> newconfig duplicates these informations is that these informations is
> needed on both dynamic linking and static linking.
> Because the new-bus puts these informations into kernel modules
> itself, these informations cannot be used on static linking. 
> The newconfig will work well with both dynamic linking and static
> linking. But new-bus works well with dynamic linking only.

No, the relationships under new-bus are built into drivers in the
DEVICE_MODULE() declaration.  It works identically between static and
dynamic linking.  It's actually keyed on ascii strings that are registered
with the configuration manager as part of the early startup of the module
and tables are built on the fly.

I suspect there might be some confusion between linker scope dependencies
with inter-kld-module dependencies, and bus/device driver dependencies.
new-bus handles device/bus driver dependencies in all forms, both static
and dynamic, and can handle new relationships if they are added on the fly.

Handling symbol table link dependencies etc is a very different problem,
and I'm not sure that newconfig generates a mechanism for building kld's
with dependency information, does it?

Anyway, I can demonstrate new-bus working statically from one of my systems
here:

[...]
npx0: <math processor> on motherboard
npx0: INT 16 interface
apm0: <APM BIOS> on motherboard
apm: found APM BIOS version 1.2
pcib0: <PCI host bus adapter> on motherboard
pci0: <PCI bus> on pcib0
chip0: <VIA 82C597 (Apollo VP3) system controller> at device 0.0 on pci0
pcib1: <VIA 82C598MVP (Apollo MVP3) PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
isab0: <VIA 82C586 PCI-ISA bridge> at device 7.0 on pci0
isa0: <ISA bus> on isab0
ide_pci0: <VIA 82C586x (Apollo) Bus-master IDE controller> at device 7.1 on pci0
uhci0: <VIA 83C572 USB Host Controller> at device 7.2 on pci0
uhci0: interrupting at irq 10
usb0: <VIA 83C572 USB Host Controller> on uhci0
uhub0 at usb0
uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
chip1: <VIA 82C586B ACPI interface> at device 7.3 on pci0
de0: <Digital 21140A Fast Ethernet> at device 11.0 on pci0
de0: interrupting at irq 9
de0: SMC 9332BDT 21140A [10-100Mb/s] pass 2.2
de0: address 00:e0:29:06:c1:e3
fdc0: interrupting at irq 6
fdc0: <NEC 72065B or clone> at port 0x3f0-0x3f7 irq 6 drq 2 on isa0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> at fdc0 drive 0
atkbdc0: <keyboard controller (i8042)> at port 0x60 on isa0
atkbd0: <AT Keyboard> on atkbdc0
atkbd0: interrupting at irq 1
vga0: <Generic ISA VGA> on isa0
[...]

> So, my first question is:
> 
> [Question 1]
> 	Is there any consensus about the following issue in "core" ?
> 		The fundamental difference about new-bus and newconfig
> 		is that the new-bus only supports dynamic linking, and
> 		the newconfig supports both static linking and dynamic 
> 		linking.
> 		We are going to new-bus, and will completely avoid
> 		static linking. (static linking of device drivers will
> 		not be able to work on FreeBSD, eventually).
> 	If there are no consensus like this until now, this issue
> 	shold be approved by "core", before deciding either the new-bus
> 	or the newconfig.

I asked about this a few hours ago.  It's still a little early for
everybody to have read their email yet.  But I will be open and say that so
far the responses have been in favour of new-bus, and that it's unfortunate
that we have to choose.  Also, this issue has been discussed a number of
times over the last three years, both within core and within the developers
groups and we decided that it wasn't worth the effort each time.  The
situation is different now though.  The last times it has been raised,
there was no newconfig inplementation, so a considerable amount of work
would be required to implement something that (in our judgement) wasn't
going to get us an awful lot since it was largely static just like the old
system.  Yes, it would have cleaned things up considerably, but on the
outside it would have looked pretty much the same.

However, now we have a situation where the work has already been done,
twice, and is ready for integration from two different sets of people.

> 	The newconfig people think that it is better to support both
> 	dynamic linking and static linking.

So do we, it is essential.  For a system under active development, there
is little choice since kld modules and the kernel are built seperately
and it is too easy to have bad problems from data structure mismatches.
But for a stable system, dynamic linking is quite a viable option.

> 	For example, if one doesn't statically link a console device
> 	to a kernel, it is difficult to find the reason when dynamic
> 	linking of the console device is failed.  And also, as a
> 	matter of fact, completely dynamic configuration is not
> 	practical for now.

This is a weakness in kld that has not been addressed yet.  The
dependencies are only file based and this is not adequate.  This is a
linker problem, not a bus/device/configuration problem as it has a much
wider scope than just devices.  For example, mfs can't be loaded
unless the ufs core is present, and there's no way for loader(8) to
know whether there is a ufs in the kernel or whether it needs to be
statically loaded.

> 	Will really FreeBSD only support dynamic linking ?
> 	I've heard that dynamic linking is a only way to go, from
> 	the newbus people, several times. But do "core" members think
> 	so, too ?
> 	As you see the current FreeBSD kernel which is almost
> 	statically linked, the static linked kernel itself is not so
> 	bad. And it works better than fully dynamic linked kernel,
> 	in same cases.

I/We see this as a step in the direction of being _able_ to support dynamic
linking effectively, not requiring it.  I doubt that there will ever be
changes that are made that will require dynamic loading without a choice.

> The next issue is how to implement dependency on the new-bus.
> 
> Once, Doug wrote the following message on freebsd-mobile.
> 
> dfr> From: Doug Rabson <dfr@nlsystems.com>
> dfr> Subject: Re: Any success with CirrusLogic 6729/6730???
> dfr> Date: Mon, 12 Apr 1999 19:20:54 +0100 (BST)
> dfr> Message-ID: <Pine.BSF.4.05.9904121917570.74823-100000@herring.nlsystems.
    com>
> dfr> Delivered-To: freebsd-mobile@freebsd.org
> dfr> 
> dfr> A split driver can be dynamically loaded by using the dependancy mechani
    sm
> dfr> on the KLD modules. Imagine a device foo which support pci and isa
> dfr> attachments. One could build three modules:
> dfr>        foo.ko          containing the core driver functionality
> dfr>        foo_isa.ko      probe/attach for isa bus
> dfr>        foo_pci.ko      probe/attach for pci bus
> dfr> 
> dfr> The two bus-specific modules would depend on the foo.ko module ensuring
> dfr> that it is loaded automatically if either bus-specific module is used.
> 
> How does the new-bus implement this ?

This is implemented by kld, not new-bus.  It is not handled very well and
we understand the problem very well.

> I have two ideas.
> 
> [a1] Put the dependency information to the kernel object file by using
>   DT_NEEDED tag in elf object.

This is how it is presently implemented for ELF, and a tag in the _DYNMAIC
structure in a.out.  The effect is the same.

> [a2] Make the foo_{isa,pci}.ko load the foo.ko.
> 
> I think the [a1] is not consistent with the new-bus policy [A].
> If the new-bus choose [a1], the dependency information should be
> placed on the out side of the kernel C source to produce linker
> options which put the dependency information to the kernel object.
> If some of the these informations should be duplicated to the
> out side of the kernel C source, what's wrong about completely
> duplicates these informations to out side of the kernel C source
> (as the newconfig)? It has real benefits, these information can 
> be used on static linking, and also can be used by other utilities
> which cannot be made on the new-bus.
> I think the new-bus policy [A] is overkill (which makes static
> linking impossible without real reasons) in this case.

Ahh, I think I understand where the confusion is coming from.  The
names used for the various objects is confusing, and I think we are
talking about different things.

Let me explain what I am talking about so that we are sure that we are
arguing about the same things. :-)

What we call a kernel module is a logical "component" of the kernel source.
It might be a filesystem, driver, bus module, emulation package, whatever.

What we call a kld module or .ko object is just a binary that is linked with
the kernel at runtime.

A kld module/.ko file can have zero or more kernel modules in it.  ufs.ko
could have the ufs, ffs, lfs, ext2fs "modules" all bundled together into
one file.

KLD does not have a decent dependency mechanism, we know this.  The whole
src/sys/modules/* tree is a temporary thing and only meant to replace
functionality lost when LKM was depreciated.  We have not yet implemented
a decent and flexible file-based dependency system.

Most likely, kld and the loader will parse the file headers to find the
internal module lists and the actual module-level dependencies.  This will
solve the problem of not knowing how the modules are packaged into files.

Again, this is not a new-bus versus newconfig configuration mechanism
issue.  It's a far bigger problem than that.

I think I understand why you want to use newconfig and the definitions
files to solve this problem though as it does know about the
interdependencies between "modules".  For example, it knows that ffs
requires ufs, and so on.  But I don't think it has the capability to do
anything with that knowledge though as it's geared more towards building a
kernel than managing seperately built file modules.

> If [a2] is the way which the new-bus choose, the functions in the
> foo_isa.ko (or foo_pci.ko) cannot directly call the functions in
> foo.ko, because the foo.ko might not be loaded when the foo_isa.ko (or
> foo_pci.ko) is loaded, then the symbols in the foo.ko which is
> refered by the foo_isa.ko will be unresolved. So, in this case, 
> the new-bus have to use the indirect call like the DEVMETHOD() 
> to call the functions in foo.ko from the functions in foo_isa.ko
> (or foo_pci.ko).
> I think this is not the better way to implement the kernel. 
> In this case, the DEVMETHOD() doesn't do anything really meaningful.
> Because both foo_isa.ko and foo.ko shares many data structures, the
> DEVMETHOD() way doesn't guarantee the consitency between foo_isa.ko
> and foo.ko. And there is other better way to guarantee this.
> Note that in this case, the new-bus cannot implement the newconfig shim.
> Because the newconfig don't have to use DEVMETHOD() for this.

We don't do that.  We presently use [a1], and we know it isn't good enough.
It will most likely be replaced with [a3] here:

[a3] kldload and loader will parse the metadata in the headers of files so
that it can maintain dependencies at runtime a level much lower than at
file level.

> The newconfig doesn't have these problems. It is consistent because
> always handles the dependency information in same way, and it can use
> either direct call or indirect call like the DEVMETHOD(), according to
> programmer's choise.

We currently support both methods.  With our existing example kld's, there
is one that has a dependency.  If you load the "coff" kld, the DT_NEEDED
tag in it's header will indicate that ibcs2.ko needs to be loaded and
both the loader and kldload will automatically load them, and any of their
dependencies etc before beginning symbol resolution.

> So, second questions is
> 
> [Question 2]
> 	Which is the way the new-bus choose? [a1]? [a2]?
> 	In both cases, the newconfig seems to be better. Isn't it?

Presently [a1], but will be replaced with something better that is based on
declarations explicitly attached to the the DECLARE_MODULE() statements or
declarations generated by a replacement build system.

> And this is the last question.
> 
> [Question 3]
> 	Do the new-bus have a plan how the old-config will be?
> 	Definitely it is most ugly part of the new-bus.

The old config is still partly used.  It generates a Makefile for the
static part of the kernel using the components that were requested and a
table that contains any specific configuration instructions that the user
has requested.  It does far less than under new-bus than it does presently
and definately is going away as soon as we can manage it.

What I want to do is replace it with something that uses a reasonably
simple ruleset based mechanism that can unify the kernel and kld build into
a single process.

> I cannot read your reply for a while, because it is 23:40pm in Japan,
> and I'll go to bed.

I feel really bad about the timing of all this.  I feel especially bad that
it's turned out that there has been so much duplicated effort.  I have only
known that there even was a newconfig project for a couple of days.

> Thanks.
> 
> P.S.
> I think the KLD is the great and the best part of the new-bus.
> The newconfig follows the new-bus in this part.

KLD is still a work in progress.  It doesn't do domain based reference
counting yet and auto load on demand and unload on timeout.

> P.P.S.
> Warner, forgive me about suddenly CC'ed this mail. :-)
> 
> P.P.P.S.
> Furuta, if I miss something major issue, please say so.

The next problem is where do we go from here?  I understand very well that
you have spent a huge amount of time and effort on your work, but we can
only choose one or the other..

If we were to guarantee to you that the new-bus code *does* do everything
that the kernel parts of newconfig do and in fact it does even more while
doing 100% dynamic configuration, would that make any difference?

A lot depends on why you have chosen go to the newconfig method.  If it was
because you needed better NetBSD source compatability so that you can
better pool your resources on things like pccard/cardbus/etc, then I would
suggest that a "compatability shim" might be sufficient.  This could
provide a fairly decent emulation of the newconfig data structures and
interfaces and yet still run right alongside the rest of the system.  It
would be difficult to make it perfect, but it would certainly be a lot
easier to work under than the present system.

What I'd like to know is what your objectives are:
- - a replacement for the REALLY BAD existing config(8) system?
- - NetBSD driver source compatability?
- - because you have a personal interest in configuration mechanisms?
- - or perhaps something else?

I suspect you started newconfig because of the first two.  If so, could an
emulation of the NetBSD driver configuration mechanisms be sufficient?

As I said earlier, it is very sad that there has been so much duplicated
work and that we have to choose..   Putting personal feelings aside, there
is no doubt that newconfig is much better than the oldconfig system, but I
really do believe the new-bus mechanism is better still from a run-time and
dynamic configuration perspective and has much more opportunity to improve
even more.

Cheers,
- -Peter



------------------------------

Return-Path: soda@sra.co.jp
Received: from sramhc.sra.co.jp (root@sramhc [133.137.20.31]) by srapc342.sra.co.jp (8.8.8/3.4W-sra) with ESMTP id XAA29898 for <soda@srapc342.sra.co.jp>; Fri, 16 Apr 1999 23:10:53 +0900 (JST)
Received: from srasvf.sra.co.jp (root@srasvf.sra.co.jp [133.137.28.2])
	by sramhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id XAA07404;
	Fri, 16 Apr 1999 23:10:49 +0900 (JST)
Received: from srapc342.sra.co.jp (srapc342 [133.137.28.111])
	by srasvf.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id XAA19527;
	Fri, 16 Apr 1999 23:10:35 +0900 (JST)
Received: (from soda@localhost) by srapc342.sra.co.jp (8.8.8/3.4W-sra) id XAA29894; Fri, 16 Apr 1999 23:10:44 +0900 (JST)
Date: Fri, 16 Apr 1999 23:10:44 +0900 (JST)
Message-Id: <199904161410.XAA29894@srapc342.sra.co.jp>
From: Noriyuki Soda <soda@sra.co.jp>
To: Peter Wemm <peter@netplex.com.au>
Cc: Noriyuki Soda <soda@sra.co.jp>, Warner Losh <imp@harmony.village.org>,
        Atsushi Furuta <furuta@sra.co.jp>, Satoshi Asami <asami@freebsd.org>
Subject: Re: new-bus/newconfig 
In-Reply-To: <19990415183252.7A7081F61@spinner.netplex.com.au>
References: <199904151449.XAA16913@srapc342.sra.co.jp>
	<19990415183252.7A7081F61@spinner.netplex.com.au>

Hi,
Forgive me about late responce for your mail. Reading and writing
English takes long time for me.

> > First of all, I think fundamental design difference between new-bus
> > and newconfig is:
> > 
> > [A] new-bus puts the informations about inter-module dependency and
> >   bus hierarchy to the kernel module itself.
> > [B] newconfig puts these informations to another place. i.e. the
> >   "files" file.
> > 
> > (Note that I didn't say a kernel configuration file, but the "files" file.)
> 
> Yes, that would be a fair statement, but simplifies the situation a lot.
> 
> A couple of key implications of these design choices as I see it are:
> 
> - newconfig takes those "files" files and generates tables which are
>   statically compiled into the kernel.  The possible relationships
>   are frozen at that point I believe.

Sure, this assertion is true in current implementation of newconfig.
In current implementation, a table generated by "files" file is
statically linked to the kernel.  Of course, we are aware of the
problem you point out, so we are now modifying newconfig to resolve
the problem.  We will add new functions, which adds/removes
parent/child relationship informations of the devices, to newconfig.

So your comment is correct for a current newconfig implementation,
but it is not correct for our newconfig design, I think.

Moreover, in our newconfig design, we think these features are required:

	* A utility that compiles "files" file for the independent
	  dynamic module, and it generates a table file of
	  device tree/dependency information of the dynamic module.
	  (I think it is not so hard because it is almost same as
	   a part of config.new(8))

	* A function to load the table before the dynamic module loading
	  is invoked. The module and the table might be provided as
	  different files.

	* A function which looks up the loaded table, and invokes
	  the function to register the parent/child relationship,
	  like devclass_add_driver() in new-bus.

> - The new-bus relationships are dynamically extendable.  You can load
>   drivers and busses with relationships that were not even heard of when
>   the kernel was compiled.

As noted as above, this also can be done in newconfig.

> As an example, suppose you built a newconfig kernel with the current
> standard set of usb and pccard driver relationships from the "files.*"
> files and you are running it happily on a system that you cannot shut down.
> Then, suppose somebody supplies you with a new device which happens to be a
> pccard adapter that runs on USB.  ie: a USB->PCCARD bridge that you plug
> into a USB port and has two pccard slots.  It is my understanding that in
> order to load a driver for this new bridge, you would be stuck since the
> configuration engine has been statically configured with 'card*' attaching at
> pcic/pci/isa/etc and not the usb bridge you want to load.

This description is not correct. The newconfig can do exactly what
the new-bus can do.

> An example of this is the PCI->cardbus bridge (cbb) which is defined in
> i386/conf/files.i386.newconf and is listed as providing cardslot and
> pcmciabus if I read it right, and that this is compiled into the
> permanently into kernel.  As I understand it, if this relation wasn't
> compiled into the kernel, then it wouldn't be possible to kldload a cbb
> driver and have it automatically provide cardbus and pcmcia services.

The newconfig will be able to dynamically load the information which is
compiled from the "files" files, so this restriction will be removed in
future.

> That is what I believe is the the most critical consequence of the
> differences between new-bus and newconfig designs.

> > >> There was also the problem of duplication of information.  Drivers
> > >> in the kernel were written to attach to specific busses and the
> > >> exact same information had to be duplicated in the user's config
> > >> file.  Some alternatives (the original config.new, for example)
> > >> were worse because they added another copy - various Makefiles and
> > >> rule files needed another set of information to say the same thing
> > >> again.
> > 
> > But I think this your description is not accurate. The reason why the
> > newconfig duplicates these informations is that these informations is
> > needed on both dynamic linking and static linking.
> > Because the new-bus puts these informations into kernel modules
> > itself, these informations cannot be used on static linking. 
> > The newconfig will work well with both dynamic linking and static
> > linking. But new-bus works well with dynamic linking only.
> 
> No, the relationships under new-bus are built into drivers in the
> DEVICE_MODULE() declaration.  It works identically between static and
> dynamic linking.  It's actually keyed on ascii strings that are registered
> with the configuration manager as part of the early startup of the module
> and tables are built on the fly.

I think this DEVICE_MODULE() is typo of DRIVER_MODULE().
Is this right?
If so, I think this description is still not accurate.
In current new-bus, the filename information stored in DRIVER_MODULE()
is duplicated in old-config's "files" file, and static linker uses the
information.  We think this duplication is not needed and should be
avoided.

In newconfig, this duplication is not needed, because the information
from "files" file are used both dynamic linking and static linking.

> I suspect there might be some confusion between linker scope dependencies
> with inter-kld-module dependencies, and bus/device driver dependencies.
> new-bus handles device/bus driver dependencies in all forms, both static
> and dynamic, and can handle new relationships if they are added on the fly.

I think this is not accurate, too.
In new-bus, static linker cannot get the bus/device driver dependencies
properly. For example, the old-config doesn't know about bus/device 
hierarchy.

> Handling symbol table link dependencies etc is a very different problem,
> and I'm not sure that newconfig generates a mechanism for building kld's
> with dependency information, does it?

Currently does not. But as noted above, the major feature of the
newconfig for dynamic linkage is it. We will implement it, and
currently new-bus lacks that feature.

> > 	The newconfig people think that it is better to support both
> > 	dynamic linking and static linking.
> 
> So do we, it is essential.  For a system under active development, there
> is little choice since kld modules and the kernel are built seperately
> and it is too easy to have bad problems from data structure mismatches.
> But for a stable system, dynamic linking is quite a viable option.

I think one of the major problem of the new-bus is the
DRIVER_MODULE().  If we will support proper static linkage, both the
filename information and the bus hierarchy information which is defined
by DRIVER_MODULE() should be able to be used from the static linker.
Currently the new-bus lacks this feature, that is
	- the filename information
		duplicated in old-config's "files" file.
		(this duplication is not needed in the newconfig)
	- the bus hierarchy information 
		cannot be accessed from the static linker.
As long as these are defined in a C source by DRIVER_MODULE(),
these informations cannot be accessed from static linker.
These informations should be placed in "files" file to be able to
accessed from both static linker and dynamic configuration, like the
newconfig does.

This causes very important difference between the new-bus and the
newconfig. In the newconfig, both boot-time/runtime dynamic
configuration and static configuration uses exactly same syntax for
device hint as below:
	de*	at pci? dev ? func ?
In the new-bus, almost same syntax with the newconfig can be used in
dynamic configuration. But the syntax cannot be used in static
configuration. This is quite inconsistent from user's viewpoint.

The newconfig provides consistent configuration syntax for both static
configuration and dynamic configuration, but the new-bus doesn't.

In summary, I think the newconfig supports better static linkage than
the new-bus, and also it can support all dynamic linking feature as
the new-bus supports.

> > 	For example, if one doesn't statically link a console device
> > 	to a kernel, it is difficult to find the reason when dynamic
> > 	linking of the console device is failed.  And also, as a
> > 	matter of fact, completely dynamic configuration is not
> > 	practical for now.
> 
> This is a weakness in kld that has not been addressed yet.  The
> dependencies are only file based and this is not adequate.  This is a
> linker problem, not a bus/device/configuration problem as it has a much
> wider scope than just devices.  For example, mfs can't be loaded
> unless the ufs core is present, and there's no way for loader(8) to
> know whether there is a ufs in the kernel or whether it needs to be
> statically loaded.

And the newconfig addresses this problem by "files" file.
I think this is the best way.

> > [a1] Put the dependency information to the kernel object file by using
> >   DT_NEEDED tag in elf object.
> 
> This is how it is presently implemented for ELF, and a tag in the _DYNMAIC
> structure in a.out.  The effect is the same.
	:
> > I think the [a1] is not consistent with the new-bus policy [A].
> > If the new-bus choose [a1], the dependency information should be
> > placed on the out side of the kernel C source to produce linker
> > options which put the dependency information to the kernel object.
> > If some of the these informations should be duplicated to the
> > out side of the kernel C source, what's wrong about completely
> > duplicates these informations to out side of the kernel C source
> > (as the newconfig)? It has real benefits, these information can 
> > be used on static linking, and also can be used by other utilities
> > which cannot be made on the new-bus.
> > I think the new-bus policy [A] is overkill (which makes static
> > linking impossible without real reasons) in this case.
> 
> Ahh, I think I understand where the confusion is coming from.  The
> names used for the various objects is confusing, and I think we are
> talking about different things.
> 
> Let me explain what I am talking about so that we are sure that we are
> arguing about the same things. :-)
> 
> What we call a kernel module is a logical "component" of the kernel source.
> It might be a filesystem, driver, bus module, emulation package, whatever.
> 
> What we call a kld module or .ko object is just a binary that is linked with
> the kernel at runtime.
> 
> A kld module/.ko file can have zero or more kernel modules in it.  ufs.ko
> could have the ufs, ffs, lfs, ext2fs "modules" all bundled together into
> one file.

Thanks for clarifications.

> Most likely, kld and the loader will parse the file headers to find the
> internal module lists and the actual module-level dependencies.  This will
> solve the problem of not knowing how the modules are packaged into files.
> 
> Again, this is not a new-bus versus newconfig configuration mechanism
> issue.  It's a far bigger problem than that.

No. This *is* new-bus versus newconfig issue.

In newconfig, this is resolved by the compiled version of the "files" file.
The newconfig uses integrated way for bus/device module dependency and
normal kernel module dependency. Because the new-bus's way is not
integrated, it have to use different mechanism for bus/device module
dependency and normal module dependency. That is overkill, IMHO.

> I think I understand why you want to use newconfig and the definitions
> files to solve this problem though as it does know about the
> interdependencies between "modules".  For example, it knows that ffs
> requires ufs, and so on.  But I don't think it has the capability to do
> anything with that knowledge though as it's geared more towards building a
> kernel than managing seperately built file modules.

When the newconfig implement dynamic linkage, this problem will be
resolved.

> We don't do that.  We presently use [a1], and we know it isn't good enough.
> It will most likely be replaced with [a3] here:
> 
> [a3] kldload and loader will parse the metadata in the headers of files so
> that it can maintain dependencies at runtime a level much lower than at
> file level.

This is what the newconfig will do. All these information is handled by
compiled format of the "files" file.

> > The newconfig doesn't have these problems. It is consistent because
> > always handles the dependency information in same way, and it can use
> > either direct call or indirect call like the DEVMETHOD(), according to
> > programmer's choise.
> 
> We currently support both methods.  With our existing example kld's, there
> is one that has a dependency.  If you load the "coff" kld, the DT_NEEDED
> tag in it's header will indicate that ibcs2.ko needs to be loaded and
> both the loader and kldload will automatically load them, and any of their
> dependencies etc before beginning symbol resolution.

BTW, the newconfig also can uses this already.

> > [Question 3]
> > 	Do the new-bus have a plan how the old-config will be?
> > 	Definitely it is most ugly part of the new-bus.
> 
> The old config is still partly used.  It generates a Makefile for the
> static part of the kernel using the components that were requested and a
> table that contains any specific configuration instructions that the user
> has requested.  It does far less than under new-bus than it does presently
> and definately is going away as soon as we can manage it.
> 
> What I want to do is replace it with something that uses a reasonably
> simple ruleset based mechanism that can unify the kernel and kld build into
> a single process.

I think replacing the old-config by newconfig is the best answer...

> > I cannot read your reply for a while, because it is 23:40pm in Japan,
> > and I'll go to bed.
> 
> I feel really bad about the timing of all this.  I feel especially bad that
> it's turned out that there has been so much duplicated effort.  I have only
> known that there even was a newconfig project for a couple of days.

I see, I feel lack of our communication.  I apologize that we lack of
our effort to explain newconfig design and the reason why we think
new-bus is not perfect. It was mostly caused by language barrier, but
anyway we are very sorry about we were not able to supply enough
information.

> The next problem is where do we go from here?  I understand very well that
> you have spent a huge amount of time and effort on your work, but we can
> only choose one or the other..
> 
> If we were to guarantee to you that the new-bus code *does* do everything
> that the kernel parts of newconfig do and in fact it does even more while
> doing 100% dynamic configuration, would that make any difference?

Mmmm, As noted above, I still think that the new-config does better
job for static linkage, and provides simple and efficient mechanism
for dynamic linkage, than the new-bus.

Major flaws of the current new-bus implementation is:

- - DRIVER_MODULE(). This information should be provided to static linker,
  too.
- - inter kld module dependency. This can be handled by same way as bus
  dependency/hierarchy handling on the newconfig. The new-bus requires
  different machanism, and it is overkill.
- - old-config. it lacks of functionary which the newconfig has,
  and is machine dependent and bus dependent.

This cause the following problem:

- - syntax of static configuration and dynamic configuration are different.

> A lot depends on why you have chosen go to the newconfig method.  If it was
> because you needed better NetBSD source compatability so that you can
> better pool your resources on things like pccard/cardbus/etc, then I would
> suggest that a "compatability shim" might be sufficient.

Probably the newconfig can provide "compatibility shim" for new-bus,
too....

> What I'd like to know is what your objectives are:
> - a replacement for the REALLY BAD existing config(8) system?
> - NetBSD driver source compatability?
> - because you have a personal interest in configuration mechanisms?
> - or perhaps something else?

I'd like to replace the following new-bus modules by the newconfig
modules, if it is acceptable, though I know this is quite hard to
accept, and must cause big pain to new-bus people. Mmmmm.

- - kern/subr_bus.c by kern/subr_autoconf.c
- - DRIVER_MODULE() by the newconfig's "files" file (and it's compiled format.)
- - the old-config by the newconfig

> As I said earlier, it is very sad that there has been so much duplicated
> work and that we have to choose..

I completely agree. I hope that there is the way we can go together.
- --
soda

------------------------------

Return-Path: peter@netplex.com.au
Received: from sramhc.sra.co.jp (root@sramhc [133.137.20.31]) by srapc342.sra.co.jp (8.8.8/3.4W-sra) with ESMTP id BAA00841 for <soda@srapc342.sra.co.jp>; Sat, 17 Apr 1999 01:05:03 +0900 (JST)
Received: from sranha.sra.co.jp (sranha.sra.co.jp [133.137.8.8])
	by sramhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id BAA08470;
	Sat, 17 Apr 1999 01:05:00 +0900 (JST)
Received: from sraigw.sra.co.jp (sraigw-hub [133.137.8.14])
	by sranha.sra.co.jp (8.8.7/3.6Wbeta7-sranha) with ESMTP id BAA02734;
	Sat, 17 Apr 1999 01:04:59 +0859 (JST)
Received: from spinner.netplex.com.au (spinner.netplex.com.au [202.12.86.3])
	by sraigw.sra.co.jp (8.8.7/3.6Wbeta7-sraigw) with ESMTP id BAA29333;
	Sat, 17 Apr 1999 01:04:44 +0900 (JST)
Received: from netplex.com.au (localhost [127.0.0.1])
	by spinner.netplex.com.au (Postfix) with ESMTP
	id D29EE1F08; Sat, 17 Apr 1999 00:04:34 +0800 (WST)
	(envelope-from peter@netplex.com.au)
X-Mailer: exmh version 2.0.2 2/24/98
To: Noriyuki Soda <soda@sra.co.jp>
Cc: Warner Losh <imp@harmony.village.org>, Atsushi Furuta <furuta@sra.co.jp>,
        Satoshi Asami <asami@freebsd.org>
Subject: Re: new-bus/newconfig 
In-reply-to: Your message of "Fri, 16 Apr 1999 23:10:44 +0900."
             <199904161410.XAA29894@srapc342.sra.co.jp> 
Date: Sat, 17 Apr 1999 00:04:33 +0800
From: Peter Wemm <peter@netplex.com.au>
Message-Id: <19990416160434.D29EE1F08@spinner.netplex.com.au>

Noriyuki Soda wrote:
> Hi,
> Forgive me about late responce for your mail. Reading and writing
> English takes long time for me.

That is ok, and I appreciate the effort.

> > > First of all, I think fundamental design difference between new-bus
> > > and newconfig is:
> > > 
> > > [A] new-bus puts the informations about inter-module dependency and
> > >   bus hierarchy to the kernel module itself.
> > > [B] newconfig puts these informations to another place. i.e. the
> > >   "files" file.
> > > 
> > > (Note that I didn't say a kernel configuration file, but the "files" file
    .)
> > 
> > Yes, that would be a fair statement, but simplifies the situation a lot.
> > 
> > A couple of key implications of these design choices as I see it are:
> > 
> > - newconfig takes those "files" files and generates tables which are
> >   statically compiled into the kernel.  The possible relationships
> >   are frozen at that point I believe.
> 
> Sure, this assertion is true in current implementation of newconfig.
> In current implementation, a table generated by "files" file is
> statically linked to the kernel.  Of course, we are aware of the
> problem you point out, so we are now modifying newconfig to resolve
> the problem.  We will add new functions, which adds/removes
> parent/child relationship informations of the devices, to newconfig.

Yes, I realize newconfig can be changed to support this, but new-bus
already does this and has been doing it since early last year.  Why
reinvent it?

> > > >> There was also the problem of duplication of information.  Drivers
> > > >> in the kernel were written to attach to specific busses and the
> > > >> exact same information had to be duplicated in the user's config
> > > >> file.  Some alternatives (the original config.new, for example)
> > > >> were worse because they added another copy - various Makefiles and
> > > >> rule files needed another set of information to say the same thing
> > > >> again.
> > > 
> > > But I think this your description is not accurate. The reason why the
> > > newconfig duplicates these informations is that these informations is
> > > needed on both dynamic linking and static linking.
> > > Because the new-bus puts these informations into kernel modules
> > > itself, these informations cannot be used on static linking. 
> > > The newconfig will work well with both dynamic linking and static
> > > linking. But new-bus works well with dynamic linking only.
> > 
> > No, the relationships under new-bus are built into drivers in the
> > DEVICE_MODULE() declaration.  It works identically between static and
> > dynamic linking.  It's actually keyed on ascii strings that are registered
> > with the configuration manager as part of the early startup of the module
> > and tables are built on the fly.
> 
> I think this DEVICE_MODULE() is typo of DRIVER_MODULE().
> Is this right?
> If so, I think this description is still not accurate.
> In current new-bus, the filename information stored in DRIVER_MODULE()
> is duplicated in old-config's "files" file, and static linker uses the
> information.  We think this duplication is not needed and should be
> avoided.

There is no filename information stored in DRIVER_MODULE() at all.  The
names are module names and have no relationship to how it might be stored
on disk, either in the core kernel or as a .ko file.

For example,

static device_method_t fxp_methods[] = {
        /* Device interface */
        DEVMETHOD(device_probe,         fxp_probe),
        DEVMETHOD(device_attach,        fxp_attach),
        DEVMETHOD(device_detach,        fxp_detach),
        DEVMETHOD(device_shutdown,      fxp_shutdown),
 
        { 0, 0 } 
};

static driver_t fxp_driver = { 
        "fxp",
        fxp_methods,
        DRIVER_TYPE_NET,
        sizeof(struct fxp_softc),
};
 
static devclass_t fxp_devclass;

DRIVER_MODULE(fxp, pci, fxp_driver, fxp_devclass, 0, 0);
              ^^^^^^^^

All that this means is that this instance of this driver is a child of a
bus called "pci", as an ascii token.  There is no filename information at
all.

As a quick test, I just did this..  I compiled a kernel without a driver
for my fxp device and then loaded it after booting.  I've also attached
a kldstat -v of that machine to show the difference between module
and file names.

# dmesg | grep fxp
# ifconfig -a
lo0: flags=8008<LOOPBACK,MULTICAST> mtu 16384
# kldload if_fxp
fxp0: <Intel EtherExpress Pro 10/100B Ethernet> at device 6.0 on pci0
fxp0: interrupting at irq 18
fxp0: Ethernet address 00:a0:c9:49:aa:d3
# ifconfig -a
lo0: flags=8008<LOOPBACK,MULTICAST> mtu 16384
fxp0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> mtu 1500
        ether 00:a0:c9:49:aa:d3 
        media: autoselect (100baseTX) status: active
        supported media: autoselect 100baseTX <full-duplex> 100baseTX 10baseT/UTP <full-duplex> 10baseT/UTP
# dmesg | grep fxp
fxp0: <Intel EtherExpress Pro 10/100B Ethernet> at device 6.0 on pci0
fxp0: interrupting at irq 18
fxp0: Ethernet address 00:a0:c9:49:aa:d3
# kldstat -v
Id Refs Address    Size     Name
 1    2 0xc0100000 1aefa4   kernel.new
        Contains modules:
                Id Name
                 1 rootbus
                 2 isa/vga
                 3 isa/sc
                 4 isa/sio
                 5 atkbdc/psm
                 6 nexus/isa
                 7 isab/isa
                 8 nexus/npx
                 9 fdc/fd
                10 isa/fdc
                11 isa/atkbdc
                12 atkbdc/atkbd
                13 nexus/pcib
                14 root/nexus
                15 pci/ign
                16 pci/vga
                17 pci/chip
                18 pci/isab
                19 pci/pcib
                20 pcib/pci
                21 mfs
                22 ufs
                23 nfs
                24 procfs
                25 if_loop
                26 shell
                27 elf
                28 aout
 2    1 0xc0a01000 5000     if_fxp.ko
        Contains modules:
                Id Name
                29 pci/fxp
# 

Note the hierarchy of modules:
                 1 rootbus
                14 root/nexus
                13 nexus/pcib
                20 pcib/pci
                29 pci/fxp
These are spread over two files (kernel.new and if_fxp.ko)

> > I suspect there might be some confusion between linker scope dependencies
> > with inter-kld-module dependencies, and bus/device driver dependencies.
> > new-bus handles device/bus driver dependencies in all forms, both static
> > and dynamic, and can handle new relationships if they are added on the fly.
> 
> I think this is not accurate, too.
> In new-bus, static linker cannot get the bus/device driver dependencies
> properly. For example, the old-config doesn't know about bus/device 
> hierarchy.

old-config doesn't know anything about new-bus, it's just building a
Makefile and passing the settings it has into the kernel, and the kernel
decodes it.

old-config (which everybody agrees, must die), will quite happily let you
build a kernel in which the hierarchy is broken.  It would let me built
an isa-only kernel with the fxp pci driver in it, even though there is no
pci bus.  It would link ok too, but the fxp driver would be useless.

> > > 	The newconfig people think that it is better to support both
> > > 	dynamic linking and static linking.
> > 
> > So do we, it is essential.  For a system under active development, there
> > is little choice since kld modules and the kernel are built seperately
> > and it is too easy to have bad problems from data structure mismatches.
> > But for a stable system, dynamic linking is quite a viable option.
> 
> I think one of the major problem of the new-bus is the
> DRIVER_MODULE().  If we will support proper static linkage, both the
> filename information and the bus hierarchy information which is defined
> by DRIVER_MODULE() should be able to be used from the static linker.
> Currently the new-bus lacks this feature, that is
> 	- the filename information
> 		duplicated in old-config's "files" file.
> 		(this duplication is not needed in the newconfig)
> 	- the bus hierarchy information 
> 		cannot be accessed from the static linker.
> As long as these are defined in a C source by DRIVER_MODULE(),
> these informations cannot be accessed from static linker.
> These informations should be placed in "files" file to be able to
> accessed from both static linker and dynamic configuration, like the
> newconfig does.

As I said above, DRIVER_MODULE() does not contain filename information.
It is used *only* at run-time for inserting the driver into the device
tree.

The new-bus mechanism deals only with bus and device driver configuration
at runtime, it has nothing to do with building the kernel and modules.
At present that job is handled (poorly) by a hacked version of old-config,
the "files*" files and the user.  If the user configures a pci driver in a
kernel without a pci bus, old-config is stupid enough to let them do it.
Just the same as it will let them configure IPFW without INET and so on.
Building the kernel and handling source dependencies goes far beyond
the scope of the new-bus runtime configuration system.

> This causes very important difference between the new-bus and the
> newconfig. In the newconfig, both boot-time/runtime dynamic
> configuration and static configuration uses exactly same syntax for
> device hint as below:
> 	de*	at pci? dev ? func ?
> In the new-bus, almost same syntax with the newconfig can be used in
> dynamic configuration. But the syntax cannot be used in static
> configuration. This is quite inconsistent from user's viewpoint.

Why should the user be forced to say de* is at pci? dev ? func ? - the
files.pci file has declared this already.  What we want to do is to be able
to say "I want a de driver" and let the kernel figure it out, regardless
of whether de is being compiled into the core static kernel or as a module.

> The newconfig provides consistent configuration syntax for both static
> configuration and dynamic configuration, but the new-bus doesn't.

Under new-bus, there is no syntax...  The user doesn't have to say anything
other than that they want it, because it's obvious at runtime within the
kernel.  That is the key difference.

> In summary, I think the newconfig supports better static linkage than
> the new-bus, and also it can support all dynamic linking feature as
> the new-bus supports.

I accept that the subr_autoconf code could be modifed to support dynamic
configuration, but it doesn't *right now*, and new-bus does (since early
last year).

> > Most likely, kld and the loader will parse the file headers to find the
> > internal module lists and the actual module-level dependencies.  This will
> > solve the problem of not knowing how the modules are packaged into files.
> > 
> > Again, this is not a new-bus versus newconfig configuration mechanism
> > issue.  It's a far bigger problem than that.
> 
> No. This *is* new-bus versus newconfig issue.

Well, we disagree then.

> In newconfig, this is resolved by the compiled version of the "files" file.
> The newconfig uses integrated way for bus/device module dependency and
> normal kernel module dependency. Because the new-bus's way is not
> integrated, it have to use different mechanism for bus/device module
> dependency and normal module dependency. That is overkill, IMHO.

In newconfig, there are two problems being resolved.  One is in-kernel
runtime configuration, using subr_autoconf.c.  The other is managing the
kernel build and handling build-time dependencies.

On the other hand, new-bus *only* deals with the kernel runtime aspects.
We could replace the kernel build with a set of makefiles and do away with
config(8) entirely if that was convenient.

> > > [Question 3]
> > > 	Do the new-bus have a plan how the old-config will be?
> > > 	Definitely it is most ugly part of the new-bus.
> > 
> > The old config is still partly used.  It generates a Makefile for the
> > static part of the kernel using the components that were requested and a
> > table that contains any specific configuration instructions that the user
> > has requested.  It does far less than under new-bus than it does presently
> > and definately is going away as soon as we can manage it.
> > 
> > What I want to do is replace it with something that uses a reasonably
> > simple ruleset based mechanism that can unify the kernel and kld build into
> > a single process.
> 
> I think replacing the old-config by newconfig is the best answer...

We disagree there.  I believe the best answer is to replace old-config (the
program) with as close to nothing as possible and let the kernel handle
configuration at runtime.

> > The next problem is where do we go from here?  I understand very well that
> > you have spent a huge amount of time and effort on your work, but we can
> > only choose one or the other..
> > 
> > If we were to guarantee to you that the new-bus code *does* do everything
> > that the kernel parts of newconfig do and in fact it does even more while
> > doing 100% dynamic configuration, would that make any difference?
> 
> Mmmm, As noted above, I still think that the new-config does better
> job for static linkage, and provides simple and efficient mechanism
> for dynamic linkage, than the new-bus.

For static linkage, yes, newconfig does a better job.  But the cost of
it is terrible..  For example, from your example patch,  here's a section
from your NEWCONF file:

mainbus0 at root

# Basic Bus Support

# PCI bus support
pci*    at mainbus? bus ?
pci*    at pchb? bus ?
pci*    at ppb? bus ?
chipset* at pci? dev ? func ?
vga*    at pci? dev ? func ?

# PCI bridges
pchb*   at pci? dev ? func ?    # PCI-Host bridges
pcib*   at pci? dev ? func ?    # PCI-ISA bridges
ppb*    at pci? dev ? func ?    # PCI-PCI bridges
cbb*    at pci? dev ? func ?    # PCI-CardBus bridges
[..]

cardbus* at cbb? slot ?
pcmcia*  at cbb? socket ?
pcmcia*  at pcic? controller ? socket ?

# ISA bus support
isa*    at mainbus?
isa*    at pcib?

pnp*    at mainbus?

pcic0   at isa? iobase 0x3e0 membase 0xd0000 memsize 0x4000
pcic1   at isa? iobase 0x3e2 membase 0xd4000 memsize 0x4000
[..]
# PCI SCSI controllers
adv0    at pci?
adw*    at pci?
ahc*    at pci?
#amd*   at pci?
bt0     at pci?
dpt*    at pci?
isp*    at pci?
ncr*    at pci?

# SCSI bus support
scbus*  at adv?
scbus*  at adw?
scbus*  at aha?
scbus*  at ahc?
#scbus* at amd?
scbus*  at bt?
scbus*  at dpt?
scbus*  at isp?
scbus*  at ncr?

# SCSI devices
da*     at scbus? target ? lun ?        # SCSI disk drives
sa*     at scbus? target ? lun ?        # SCSI tape drives
cd*     at scbus? target ? lun ?        # SCSI CD-ROM drives
pass*   at scbus? target ? lun ?        # CAM passthrough driver
[..]
# PCI network interfaces

ax*     at pci? dev ? func ?
de*     at pci? dev ? func ?
ed0     at pci? dev ? func ?
#ed*    at pci? dev ? func ?
en*     at pci? dev ? func ?
fpa*    at pci? dev ? func ?
[..]

Ok, this is from the file that the user has to write.  Note the duplicate
information:

files.i386.newconf:

define  mainbus { }
#device mainbus at root class dull: isabus, eisabus, pcibus, mainbus
device  mainbus: isabus, pnpbus, eisabus, pcibus, mainbus
attach  mainbus at root

#device isa class dull
device  isa  {[iobase = -1], [iosize = 0],
             [membase = 0], [memsize = 0],
             [irq = -1], [drq = -1], [drq2 = -1], [flag = 0],
             [enable = 1], [imask = ISA_IMASK_NONE]}
device  isapnp {[port = -1], [size = 0],
             [iomem = -1], [iosiz = 0],
             [irq = -1], [drq = -1]}

attach  isa at isabus

So far, the user has had to specify the mainbus at root, that is pretty
obvious.  The user has had to duplicate declaring all the pci busses etc as
is already explicitly specified in the file.i386.newconf.  They have also
had to specify where all the drivers attach to, even though that is also in
the files.i386.newconf file.

conf/files.newconf:

define  isabus { }                      # ISA attachment
define  pnpbus { }                      # ISA PnP attachment
define  eisabus { }                     # EISA attachment
define  pcibus {[bus = -1]}             # PCI attachment
define  usbus { }                       # USB attachment
define  pcmciabus { [controller = -1], [socket = -1]}   # PCMCIA bus attachment
define  cardslot {[slot= -1]}           # CardBus attachment

#device pci class dull {[dev = -1], [func = -1]}
device  pci {[dev = -1], [func = -1]}
attach  pci at pcibus

device  ppb: pcibus
attach  ppb at pci

device  pchb: pcibus
attach  pchb at pci
[..]
device  pass
attach  pass at scbus

device  aha: scsi

device  ahc: scsi
attach  ahc at pci with ahc_pci

device  amd: scsi
attach  amd at pci

device  adv: scsi
attach  adv at pci with adv_pci

device  adw: scsi
attach  adw at pci with adw_pci

device  bt: scsi
attach  bt at pci with bt_pci

device  de: ether, ifnet
attach  de at pci

device  ed: ether, ifnet
attach  ed at pci with ed_pci

device  en: atm, ifnet
attach  en at pci with en_pci
[..]

Ok, so files.newconf has explicitly stated that de attaches at pci.  Why
does the user have to repeat it in their kernel config file?

The kernel provides a whole host of information in those files.*.newconf
files, and yet the user is apparently forced to supply it too..

> Major flaws of the current new-bus implementation is:
> 
> - DRIVER_MODULE(). This information should be provided to static linker,
>   too.

DRIVER_MODULE() is not used for either static or dynamic linking, it's soley
for the bus hierarchy.

> - inter kld module dependency. This can be handled by same way as bus
>   dependency/hierarchy handling on the newconfig. The new-bus requires
>   different machanism, and it is overkill.

Inter-kld module dependency will be a huge problem for newconfig too since
the linker only knows about file scope dependencies and nothing about
dependencies on what is inside them.  If 'if_fxp.ko' is dependent on the
pci code and it's not in the current kernel, the loader(8) and kld have *no
idea* what file to get it from.

> - old-config. it lacks of functionary which the newconfig has,
>   and is machine dependent and bus dependent.

Actually, the old-config code no longer has any machine dependent code at
all, as it has no knowledge of bus/device relationships. 

> This cause the following problem:
> 
> - syntax of static configuration and dynamic configuration are different.

Well, they are presently built seperately.  If the newconfig user program
was able to manage this, it would have an advantage.  I don't know if it
does because I have not been able to see it.  I fetched your newconfig
patch for the kernel, but I can't find the newconfig program to drive it.

> > A lot depends on why you have chosen go to the newconfig method.  If it was
> > because you needed better NetBSD source compatability so that you can
> > better pool your resources on things like pccard/cardbus/etc, then I would
> > suggest that a "compatability shim" might be sufficient.
> 
> Probably the newconfig can provide "compatibility shim" for new-bus,
> too....

Start with supporting smbus, iicbus and ppbus for examples of what you'd need
to get working. :-)

> > What I'd like to know is what your objectives are:
> > - a replacement for the REALLY BAD existing config(8) system?
> > - NetBSD driver source compatability?
> > - because you have a personal interest in configuration mechanisms?
> > - or perhaps something else?
> 
> I'd like to replace the following new-bus modules by the newconfig
> modules, if it is acceptable, though I know this is quite hard to
> accept, and must cause big pain to new-bus people. Mmmmm.
>
> - kern/subr_bus.c by kern/subr_autoconf.c
> - DRIVER_MODULE() by the newconfig's "files" file (and it's compiled format.)
> - the old-config by the newconfig

To be quite honest, I don't think that's going to happen, as we (freebsd
core and the rest of the developers involved in the area) feel that
newconfig doesn't take us in the direction that we want to go.  Many of us
find the syntax of the config and files.* files rather awful painfully
verbose.  Also, our Alpha port is dependant on the new-bus mechanism, so a
change to newconfig will break it.
 
> > As I said earlier, it is very sad that there has been so much duplicated
> > work and that we have to choose..
>
> I completely agree. I hope that there is the way we can go together.

A similar thing is coming up, regarding IPv6 stacks.  We are going to have
to choose sometime soon..  No matter which one gets chosen, not everybody
can be happy about the eventual choice.

Cheers,
- -Peter


------------------------------

Return-Path: soda@sra.co.jp
Received: from sramhc.sra.co.jp (root@sramhc [133.137.20.31]) by srapc342.sra.co.jp (8.8.8/3.4W-sra) with ESMTP id DAA01338 for <soda@srapc342.sra.co.jp>; Sat, 17 Apr 1999 03:31:00 +0900 (JST)
Received: from srasvf.sra.co.jp (root@srasvf.sra.co.jp [133.137.28.2])
	by sramhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id DAA09905;
	Sat, 17 Apr 1999 03:30:56 +0900 (JST)
Received: from srapc342.sra.co.jp (srapc342 [133.137.28.111])
	by srasvf.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id DAA21505;
	Sat, 17 Apr 1999 03:30:40 +0900 (JST)
Received: (from soda@localhost) by srapc342.sra.co.jp (8.8.8/3.4W-sra) id DAA01333; Sat, 17 Apr 1999 03:30:51 +0900 (JST)
Date: Sat, 17 Apr 1999 03:30:51 +0900 (JST)
Message-Id: <199904161830.DAA01333@srapc342.sra.co.jp>
From: Noriyuki Soda <soda@sra.co.jp>
To: Peter Wemm <peter@netplex.com.au>
Cc: Noriyuki Soda <soda@sra.co.jp>, Warner Losh <imp@harmony.village.org>,
        Atsushi Furuta <furuta@sra.co.jp>, Satoshi Asami <asami@freebsd.org>
Subject: Re: new-bus/newconfig 
In-Reply-To: <19990416160434.D29EE1F08@spinner.netplex.com.au>
References: <199904161410.XAA29894@srapc342.sra.co.jp>
	<19990416160434.D29EE1F08@spinner.netplex.com.au>

>>>>> On Sat, 17 Apr 1999 00:04:33 +0800,
	Peter Wemm <peter@netplex.com.au> said:

> Yes, I realize newconfig can be changed to support this, but new-bus
> already does this and has been doing it since early last year.  Why
> reinvent it?

Because
- - the newconfig is earlier than the new-bus. it is the 4.4BSD standard
  configuration program, and freely available since at least 1994.
- - the newconfig supports static configuration better than the new-bus.
- - the newconfig is fully compatible with other BSDs, the new-bus is not.

>> I think this DEVICE_MODULE() is typo of DRIVER_MODULE().
>> Is this right?
>> If so, I think this description is still not accurate.
>> In current new-bus, the filename information stored in DRIVER_MODULE()
>> is duplicated in old-config's "files" file, and static linker uses the
>> information.  We think this duplication is not needed and should be
>> avoided.

> There is no filename information stored in DRIVER_MODULE() at all.  The
> names are module names and have no relationship to how it might be stored
> on disk, either in the core kernel or as a .ko file.

Oh, then I confused something about new-bus.
Thanks for pointing out it.
But that means rather bad for new-bus. (see below.)

>> This causes very important difference between the new-bus and the
>> newconfig. In the newconfig, both boot-time/runtime dynamic
>> configuration and static configuration uses exactly same syntax for
>> device hint as below:
>> de*	at pci? dev ? func ?
>> In the new-bus, almost same syntax with the newconfig can be used in
>> dynamic configuration. But the syntax cannot be used in static
>> configuration. This is quite inconsistent from user's viewpoint.

> Why should the user be forced to say de* is at pci? dev ? func ? - the
> files.pci file has declared this already.

Because the user have a choice to whether the "de" driver should be
linked or not, in both static configuration and dynamic configuration
case. (see below [b])

>  What we want to do is to be able to say "I want a de driver" and
> let the kernel figure it out, regardless of whether de is being
> compiled into the core static kernel or as a module.

Then, on new-bus, how can the static/dynamic linker to find the
filename of the kernel module ?
Probably the user has to know the filename of the module.
Doesn't he?

In the newconfig, the user only has to specify "de* at pci?",
then both dynamic linker and static linker can link the object,
automatically.

In the new-bus, the user should know both 
	- the object filename (if_de.ko).
and
	- the device name ("de0 at pci0 slot 7")
	  when he have to wire down the device name.

This is quite problematic, think about that newer kernel split if_de.ko
to if_de.ko (bus independent backend) and if_de_pci.ko (pci bus
dependent frontend), then suddenly old setting file for dynamic
configuration becomes to fail, because the frontend driver object
name is changed from if_de.ko to if_de_pci.ko.

The newconfig doesn't have this problem, because the filename meta
data is stored in another file, and setting file only have to include
the information of the device name ("de* at pci?"). 

The configuration file should not include implementation (actual
kernel module name), but only specification (device name).
The way of the new-bus makes kernel-version-up painful.

>> The newconfig provides consistent configuration syntax for both static
>> configuration and dynamic configuration, but the new-bus doesn't.

> Under new-bus, there is no syntax...  The user doesn't have to say anything
> other than that they want it, because it's obvious at runtime within the
> kernel.  That is the key difference.

No. I think it is not accurate.
The new-bus have the syntax for specifing the configuration hint.
(e.g. the information like "controller ppc0 at isa? port ? tty irq 7")

The new-bus only use this information for configuration hint.
The newconfig uses this information for both configuration hint and
what device should be linked (both dynamic and static case in same
syntax).

As noted above, the newconfig way is better.

>> In summary, I think the newconfig supports better static linkage than
>> the new-bus, and also it can support all dynamic linking feature as
>> the new-bus supports.

> I accept that the subr_autoconf code could be modifed to support dynamic
> configuration, but it doesn't *right now*, and new-bus does (since early
> last year).

But the new-bus heavily depends on the old-config which should be
avoided. And as written in previous mail, we don't have to hurry to
switch to dynamic configuration. We need static configuration anyway,
for now. And also, we are already working to make newconfig dynamic.
It is not completed, yet. But it will come soon.

So, IMHO, the decision should be made by which has flexible and
consistent architecture. 
And I think that the newconfig is:
- - more flexible
	The newconfig supports both static configuration and the 
	dynamic configuration.
- - more consistent
	Exactly same information can be used for both static
	and dynamic configuration.
	It uses the "files" file for dependency information
	always (both static and dynamic configuration).
	The new-bus currently lacks the feature to handle 
	dependency.
- - easy to maintenance
	The dynamic loading configuration doesn't depend on the
	drivers' implementation (i.e. object filename), but the
	driver's specification (i.e. device name).

>> > Most likely, kld and the loader will parse the file headers to find the
>> > internal module lists and the actual module-level dependencies.  This will
>> > solve the problem of not knowing how the modules are packaged into files.
>> > 
>> > Again, this is not a new-bus versus newconfig configuration mechanism
>> > issue.  It's a far bigger problem than that.
>> 
>> No. This *is* new-bus versus newconfig issue.

> Well, we disagree then.

In either case, the newconfig have the framework to handle the
dependency, and this could be used by both static and dynamic
configuration.
And the new-bus currently lacks this feature.

>> In newconfig, this is resolved by the compiled version of the "files" file.
>> The newconfig uses integrated way for bus/device module dependency and
>> normal kernel module dependency. Because the new-bus's way is not
>> integrated, it have to use different mechanism for bus/device module
>> dependency and normal module dependency. That is overkill, IMHO.

> In newconfig, there are two problems being resolved.  One is in-kernel
> runtime configuration, using subr_autoconf.c.  The other is managing the
> kernel build and handling build-time dependencies.

> On the other hand, new-bus *only* deals with the kernel runtime aspects.
> We could replace the kernel build with a set of makefiles and do away with
> config(8) entirely if that was convenient.

In the newconfig,
- - the filename information in "files" file can be used by both
	static kernel building
  and
	dynamic kernel module loading
- - the dependency information in "files" file can be used by both
	static kernel building
  and
	dynamic kernel module loading.

In the new-bus,
- - there are no information about the file name in framework, so
	in static kernel building
		the filenames information scatters in various
		Makefiles.
		(currently this is supported by old-config)
	in dynamic kernel module loading
		the user has to know the filename. (not device name)
- - Also, dependency information is not actually supported in framework, 
  so
	in static kernel building
		to be determined in future.
		(currently this is supported by old-config)
	in dynamic kernel module loading
		to be determined in future.

I think it is obvious that the newconfig is better than new-bus,
in this point, too.

>> Mmmm, As noted above, I still think that the new-config does better
>> job for static linkage, and provides simple and efficient mechanism
>> for dynamic linkage, than the new-bus.

> For static linkage, yes, newconfig does a better job.  But the cost of
> it is terrible..  For example, from your example patch,  here's a section
> from your NEWCONF file:

It is not terrible.
This files includes the following informations:
- - what device driver should be loaded (both static and dynamic case)
- - configuration hints (esp. for ISA devices)
- - sample for wiring down the device name (for PCI and other modern buses)
							-- [b]
For these reasons, the new-bus should have to document exactly same
informations as the newconfig, plus, the new-bus have to document the
name of kernel module.

So, theoretically, the newconfig case is shorter than the new-bus
case, from user's point of view.

> Ok, this is from the file that the user has to write.  Note the duplicate
> information:

This is NOT the duplicate information.
These information is exactly same as the information written in the
DRIVER_MODULE() macros in the new-bus.

Note that the new-bus doesn't have a feature that describe the
dependency of the modules, so the newconfig case is longer in this
point.
But this is not problem, this simply means the new-bus doesn't have
this feature, yet.

And as already noted, there is the reason we have to split these
informations from the kernel C source (like the new-bus does).

> So far, the user has had to specify the mainbus at root, that is pretty
> obvious.

And it harms nothing. BSD/OS, NetBSD, OpenBSD users doesn't meet
trouble about this :-)

>   The user has had to duplicate declaring all the pci busses etc as
> is already explicitly specified in the file.i386.newconf.  

Because there are reasons it should be specified, please look at [b].
(which is described above)

> They have also had to specify where all the drivers attach to, even
> though that is also in the files.i386.newconf file.

Of course! The bus where the drivers attached to should be specified,
definitely. This is key part of the separation of the bus dependent
part driver and bus independent part driver. If you don't specify the
parent bus, the configuation mechanism cannot know which bus dependent
frontend driver should be used.

> Ok, so files.newconf has explicitly stated that de attaches at pci.  Why
> does the user have to repeat it in their kernel config file?

Because there is the CardBus based "de" device!

If you eliminate the parent bus name ("at pci"), then,
as soon as the CardBus "de" driver is imported,
old configuration file fails to work.
If you want to make the configuration file work between kernel
updates, the parent bus name should be specifed always.

The elimination of the parent bus name cause compatibility problem.

> The kernel provides a whole host of information in those files.*.newconf
> files, and yet the user is apparently forced to supply it too..

As noted above, this is misunderstanding. All informations of the
newconfig is mandatory.

>> Major flaws of the current new-bus implementation is:
>> 
>> - DRIVER_MODULE(). This information should be provided to static linker,
>> too.

> DRIVER_MODULE() is not used for either static or dynamic linking, it's soley
> for the bus hierarchy.

And the bus hierarchy information is needed by the staic linking,
to specify the parent bus of the driver.

Defining this information is the C source makes this information
cannot be accessed from the static linker. That's one of the most
worst part of the new-bus design.

>> - inter kld module dependency. This can be handled by same way as bus
>> dependency/hierarchy handling on the newconfig. The new-bus requires
>> different machanism, and it is overkill.

> Inter-kld module dependency will be a huge problem for newconfig too since
> the linker only knows about file scope dependencies and nothing about
> dependencies on what is inside them.  If 'if_fxp.ko' is dependent on the
> pci code and it's not in the current kernel, the loader(8) and kld have *no
> idea* what file to get it from.

No. This is not the problem. Because there are no need to combine each
*.o to one *.ko in the newconfig, the loader knows the all *.o name
from the compiled version of the "files" file, and can load all *.o
file all it once.

And, also, there is the way to specify "*.ko" filename and it's
dependency explicity in config.new(8). The 4.4BSD config.new(8)
doesn't have this feature, but NetBSD based config.new(8) has this.

So, all dependency information can be handled by "files" file,
flawlessly. The new-bus doesn't have this feature, yet.

>> - old-config. it lacks of functionary which the newconfig has,
>> and is machine dependent and bus dependent.

> Actually, the old-config code no longer has any machine dependent code at
> all, as it has no knowledge of bus/device relationships. 

And I think then old-config is wrong. To use same kernel configuration
file in both static and dynamic linkage, the config program should
handle bus/device dependency (i.e. "xxx* at xxx?")

>> This cause the following problem:
>> 
>> - syntax of static configuration and dynamic configuration are different.

> Well, they are presently built seperately.  If the newconfig user program
> was able to manage this, it would have an advantage.  I don't know if it
> does because I have not been able to see it.  I fetched your newconfig
> patch for the kernel, but I can't find the newconfig program to drive it.

Then we agree that this is advantage of the newconfig. Thanks.

The config.new(8) can be retrieved via CVSup.
More information is written is the following WWW page:
	http://www.jp.freebsd.org/newconfig/cvs.html

>> Probably the newconfig can provide "compatibility shim" for new-bus,
>> too....

> Start with supporting smbus, iicbus and ppbus for examples of what you'd need
> to get working. :-)

IMHO, I think these buses are rather minor.
What we have to handle now is the PCCard and CardBus.
The newconfig has advantage in these area.

>> I'd like to replace the following new-bus modules by the newconfig
>> modules, if it is acceptable, though I know this is quite hard to
>> accept, and must cause big pain to new-bus people. Mmmmm.
>> 
>> - kern/subr_bus.c by kern/subr_autoconf.c
>> - DRIVER_MODULE() by the newconfig's "files" file
>>   (and it's compiled format.)
>> - the old-config by the newconfig

> To be quite honest, I don't think that's going to happen, as we (freebsd
> core and the rest of the developers involved in the area) feel that
> newconfig doesn't take us in the direction that we want to go.  Many of us
> find the syntax of the config and files.* files rather awful painfully
> verbose.

I think this is because they don't understand newconfig correctly.
As written in this mail, there are many misunderstandings what
the newconfig is.

> Also, our Alpha port is dependant on the new-bus mechanism, so a
> change to newconfig will break it.

FreeBSD/alpha is derived from NetBSD/alpha, and NetBSD/alpha uses
the newconfig, so theoretically there are no reasons we cannot 
use the newconfig on FreeBSD/alpha.

And, also perhaps "new-bus shim on newconfig" can be build.

>> > As I said earlier, it is very sad that there has been so much duplicated
>> > work and that we have to choose..
>> 
>> I completely agree. I hope that there is the way we can go together.

> A similar thing is coming up, regarding IPv6 stacks.  We are going to have
> to choose sometime soon..  No matter which one gets chosen, not everybody
> can be happy about the eventual choice.

I wish that FreeBSD core does correct decision.

IMHO, the newconfig has better design than the new-bus in many points,
especially consistent handling of both static and dynamic configuration.
The new-bus lacks this feature.

Of course I'm biased, but I have not found the real point where the
new-bus is better than the newconfig, yet. If you know such point,
please tell me. If there is real reason to select the new-bus, I can
understand any decision.
- --
soda

------------------------------

Return-Path: soda@sra.co.jp
Received: from sramhc.sra.co.jp (root@sramhc [133.137.20.31]) by srapc342.sra.co.jp (8.8.8/3.4W-sra) with ESMTP id RAA07911 for <soda@srapc342.sra.co.jp>; Sat, 17 Apr 1999 17:49:52 +0900 (JST)
Received: from srasvf.sra.co.jp (root@srasvf.sra.co.jp [133.137.28.2])
	by sramhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id RAA13552;
	Sat, 17 Apr 1999 17:49:50 +0900 (JST)
Received: from srapc342.sra.co.jp (srapc342 [133.137.28.111])
	by srasvf.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id RAA22958;
	Sat, 17 Apr 1999 17:49:35 +0900 (JST)
Received: (from soda@localhost) by srapc342.sra.co.jp (8.8.8/3.4W-sra) id RAA07907; Sat, 17 Apr 1999 17:49:45 +0900 (JST)
Date: Sat, 17 Apr 1999 17:49:45 +0900 (JST)
Message-Id: <199904170849.RAA07907@srapc342.sra.co.jp>
From: Noriyuki Soda <soda@sra.co.jp>
To: Peter Wemm <peter@netplex.com.au>
Cc: Noriyuki Soda <soda@sra.co.jp>, Warner Losh <imp@harmony.village.org>,
        Atsushi Furuta <furuta@sra.co.jp>, Satoshi Asami <asami@freebsd.org>
Subject: Re: new-bus/newconfig 
In-Reply-To: <199904161830.DAA01333@srapc342.sra.co.jp>
References: <199904161410.XAA29894@srapc342.sra.co.jp>
	<19990416160434.D29EE1F08@spinner.netplex.com.au>
	<199904161830.DAA01333@srapc342.sra.co.jp>

>>>>> On Sat, 17 Apr 1999 00:04:33 +0800,
> 	Peter Wemm <peter@netplex.com.au> said:

>> A similar thing is coming up, regarding IPv6 stacks.  We are going to have
>> to choose sometime soon..  No matter which one gets chosen, not everybody
>> can be happy about the eventual choice.

Hmm, new-bus/i386 seems to be imported.
Do the core members officially decide to new-bus ?

If so, may I post this discussion to public list as a record ?

And,

>>>>> On Sat, 17 Apr 1999 03:30:51 +0900 (JST),
	Noriyuki Soda <soda@sra.co.jp> said:
> but I have not found the real point where the new-bus is better than
> the newconfig, yet. If you know such point, please tell me.

As far as I can see, there is no point where the new-bus is better
than the newconfig in it's design. Is this correct ?
- --
soda

------------------------------

Return-Path: peter@netplex.com.au
Received: from sramhc.sra.co.jp (root@sramhc [133.137.20.31]) by srapc342.sra.co.jp (8.8.8/3.4W-sra) with ESMTP id TAA08254 for <soda@srapc342.sra.co.jp>; Sat, 17 Apr 1999 19:51:06 +0900 (JST)
Received: from sranha.sra.co.jp (sranha.sra.co.jp [133.137.8.8])
	by sramhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id TAA14608;
	Sat, 17 Apr 1999 19:51:04 +0900 (JST)
Received: from sraigw.sra.co.jp (sraigw-hub [133.137.8.14])
	by sranha.sra.co.jp (8.8.7/3.6Wbeta7-sranha) with ESMTP id TAA19129;
	Sat, 17 Apr 1999 19:51:02 +0900 (JST)
Received: from spinner.netplex.com.au (spinner.netplex.com.au [202.12.86.3])
	by sraigw.sra.co.jp (8.8.7/3.6Wbeta7-sraigw) with ESMTP id TAA24810;
	Sat, 17 Apr 1999 19:50:58 +0900 (JST)
Received: from netplex.com.au (localhost [127.0.0.1])
	by spinner.netplex.com.au (Postfix) with ESMTP
	id 49C511F5D; Sat, 17 Apr 1999 18:50:50 +0800 (WST)
	(envelope-from peter@netplex.com.au)
X-Mailer: exmh version 2.0.2 2/24/98
To: Noriyuki Soda <soda@sra.co.jp>
Cc: Warner Losh <imp@harmony.village.org>, Atsushi Furuta <furuta@sra.co.jp>,
        Satoshi Asami <asami@freebsd.org>
Subject: Re: new-bus/newconfig 
In-reply-to: Your message of "Sat, 17 Apr 1999 17:49:45 +0900."
             <199904170849.RAA07907@srapc342.sra.co.jp> 
Date: Sat, 17 Apr 1999 18:50:50 +0800
From: Peter Wemm <peter@netplex.com.au>
Message-Id: <19990417105052.49C511F5D@spinner.netplex.com.au>

Noriyuki Soda wrote:
> >>>>> On Sat, 17 Apr 1999 00:04:33 +0800,
> > 	Peter Wemm <peter@netplex.com.au> said:
> 
> >> A similar thing is coming up, regarding IPv6 stacks.  We are going to have
> >> to choose sometime soon..  No matter which one gets chosen, not everybody
> >> can be happy about the eventual choice.
> 
> Hmm, new-bus/i386 seems to be imported.
> Do the core members officially decide to new-bus ?

Core reached consensis for the most part two days ago, and apon completion
of the userconfig patches, it was reconfirmed - but with great regret that
you and I were unable to resolve our differences.

> If so, may I post this discussion to public list as a record ?

I was discussing with you as a developer, not as an official core
discussion.  I wrote with the understanding that it was between your group
and me, for that reason I'd prefer not to post it outside your group.
However, I am quite happy to discuss with you in a public forum the
relative merits and weaknesses of each system, but please keep in mind that
we do have specific reasons and goals in mind behind our decisions, both
over the last few days and as far back as several years ago when config.new
was specifically decided against.

> And,
> 
> >>>>> On Sat, 17 Apr 1999 03:30:51 +0900 (JST),
> 	Noriyuki Soda <soda@sra.co.jp> said:
> > but I have not found the real point where the new-bus is better than
> > the newconfig, yet. If you know such point, please tell me.
> 
> As far as I can see, there is no point where the new-bus is better
> than the newconfig in it's design. Is this correct ?
> --
> soda

We definately disagree on a good number of points in the previous
discussions we have had.

Let me try and explain our goals:

1: kernel configuration is presently too hard.  Too many people have enough
   trouble with a kernel config file at all.  It needs to be made easier
   and not more complicated.  To my mind (and many others) the config.new
   file format and syntax requirements are excessive.

2: A good deal of automatic configuration is possible at runtime.  We have
   been talking about the de driver in the past, lets use that as an example.
   Under newconfig, you claim that it is necessary for the user to specify
   that de* goes on the pci bus, or it will break when/if a de device appears
   on the cardbus.  I dispute this by counter example.  If the de driver is
   present in the kernel, it should attach to all the busses etc that it
   knows about.  Under new-bus, the de driver would find and attach to pci
   if it was present, and would also attach to cardbus if the cardbus
   awareness was added to de.  Without the cardbus awareness in the de driver,
   nothing happens, but nothing breaks either.  Under new-bus, if the user
   then supplied a cardbus-specific attachment as a kld, it gets automatically
   re-probed and attached with no configuration changes required at all.

3: we want to better support seperate compilation where 3rd parties can build
   modules that can interface cleanly even if the new-bus configuration
   interface has changed to have more object methods.

Now, let me see if I understand your counter-claims:

1: In the de* case, suppose the driver was split into three parts.  The
   generic part without bus dependent parts, a pci-specific component and
   a cardbus specific component.  Presumably these could be built as
   seperate kld's.  I believe your argument is that by specifying everything
   completely and specifically, it will allow the run-time autoconfig
   "engine" to have the information that was needed so that it could know
   that it had to dynamically load the if_de_cardbus.ko kld to complete the
   dyanmic configuration process.  By the same argument, you believe that
   in a static kernel it will be easy to leave out the de-on-cardbus code
   if cardbus isn't present and therefore do not waste memory for it.

2: You believe it is better to put module dependency information in
   files.* rather than in the source, and that config.new(8) can generate
   the metadata to be compiled into the kernel and the kld's.

3: You believe that the run-time components (subr_autoconf.c and so on)
   can be easily extended to allow new relationships to be introduced
   apon loading even though the kernel itself was not compiled with them.

4: Using config.new is convenient because the other BSD's use it and it
   would help resolve some of the differences in drivers etc.

Is that a fair summary of your position?  To save some time, I'd like to
suggest some responses to the points above..

1: Yes, config.new has a far better implementation than our old config(8)
   for this sort of thing.  Our plan all along has been to discard
   config(8) as soon as possible and replace it with a tool specifically
   designed to be kld and loader(8) aware, using tables loaded at boot time
   from a text file for initial hints for device id -> driver matches.

2: This is a matter of preference..  It ends up in the same place.  While
   the newconfig/config.new method probably saves some repetition, it doesn't
   have the same flexibility that the in-source free-form method has.

3: I do not doubt that you could extend the kernel components to be dynamic,
   but I suspect it would be a lot of work.

4: Yes, it would help.  But there are a lot of other differences that would
   still make a fairly major task of decent driver portability.  I strongly
   suspect that the changes required to make decent dynamic configurability
   and load/unload etc support that we would require would damage this very
   same portability goal.

At the end of the day, based on what is available *right now*, I still
believe that the new-bus mechanism was the better choice.  It was designed
from the very first to be fully extensible, and while there is not an awful
lot of difference between new-bus and newconfig right now, new-bus is
designed to be easily enhanced.  I do not believe that newconfig could be
enhanced as effectively, at least not without damaging driver portabilty.

I'm terribly sorry that there has been so much duplication of effort, and I
wish that things had not worked out so that a choice had to be made.
'new-bus' had been planned as the successor to config(8) for FreeBSD for
several years and I wish that communication had been better and clearer so
that this problem had not happened.

I expect that you are bitterly disappointed, and for that I sincerely
apologize.  I do hope that your efforts on enhancing the drivers for
better configuration so far can be salvaged and I hope very much that you
can and will at some time consider working with us in the future..

Cheers,
- -Peter


------------------------------

Return-Path: soda@sra.co.jp
Received: from sramhc.sra.co.jp (root@sramhc [133.137.20.31]) by srapc342.sra.co.jp (8.8.8/3.4W-sra) with ESMTP id EAA05768 for <soda@srapc342.sra.co.jp>; Fri, 23 Apr 1999 04:22:40 +0900 (JST)
Received: from srasvf.sra.co.jp (root@srasvf.sra.co.jp [133.137.28.2])
	by sramhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id EAA13411;
	Fri, 23 Apr 1999 04:22:35 +0900 (JST)
Received: from srapc342.sra.co.jp (srapc342 [133.137.28.111])
	by srasvf.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id EAA08059;
	Fri, 23 Apr 1999 04:22:21 +0900 (JST)
Received: (from soda@localhost) by srapc342.sra.co.jp (8.8.8/3.4W-sra) id EAA05764; Fri, 23 Apr 1999 04:22:32 +0900 (JST)
Date: Fri, 23 Apr 1999 04:22:32 +0900 (JST)
Message-Id: <199904221922.EAA05764@srapc342.sra.co.jp>
From: Noriyuki Soda <soda@sra.co.jp>
To: Peter Wemm <peter@netplex.com.au>
Cc: Noriyuki Soda <soda@sra.co.jp>, Warner Losh <imp@harmony.village.org>,
        Atsushi Furuta <furuta@sra.co.jp>, Satoshi Asami <asami@freebsd.org>
Subject: Re: new-bus/newconfig 
In-Reply-To: <19990417105052.49C511F5D@spinner.netplex.com.au>
References: <199904170849.RAA07907@srapc342.sra.co.jp>
	<19990417105052.49C511F5D@spinner.netplex.com.au>

I'm sorry for quite late response.

> > If so, may I post this discussion to public list as a record ?
> 
> I was discussing with you as a developer, not as an official core
> discussion.  I wrote with the understanding that it was between your group
> and me, for that reason I'd prefer not to post it outside your group.

OK. I will not post this discussion to public list.

But the newconfig mailing list itself is public mailing list, anyone
can subscribe it, and the archive of it is accessable/searchable from
WWW. (see http://www.jp.freebsd.org/newconfig/)

How do I report this discussion to the newconfig mailing list members ?
Perhaps, is it OK to send this to each member of mailing list ?
(to prevent this discussion from being included in archive)

> We definately disagree on a good number of points in the previous
> discussions we have had.
> 
> Let me try and explain our goals:
> 
> 1: kernel configuration is presently too hard.  Too many people have enough
>    trouble with a kernel config file at all.  It needs to be made easier
>    and not more complicated.  To my mind (and many others) the config.new
>    file format and syntax requirements are excessive.

The newconfig group once already discussed this issue.
In the newconfig, the utility like Linux's "make menuconfig" can be
made and can be able to used for both static and dynamic configuration.
It must work better than the Linux "make menuconfig", because it can
retrieve the bus/device hierarchy from "files" file.
In the new-bus, such utility cannot be made for neither static nor
dynamic configuration.

> 2: A good deal of automatic configuration is possible at runtime.  We have
>    been talking about the de driver in the past, lets use that as an example.
>    Under newconfig, you claim that it is necessary for the user to specify
>    that de* goes on the pci bus, or it will break when/if a de device appears
>    on the cardbus.  I dispute this by counter example.  If the de driver is
>    present in the kernel, it should attach to all the busses etc that it
>    knows about.  Under new-bus, the de driver would find and attach to pci
>    if it was present, and would also attach to cardbus if the cardbus
>    awareness was added to de.  Without the cardbus awareness in the
>    de driver, nothing happens, but nothing breaks either.  Under
>    new-bus, if the user then supplied a cardbus-specific attachment
>    as a kld, it gets automatically re-probed and attached with no
>    configuration changes required at all.

This is also what the newconfig does in dynamic configuration.
The dynamic configuration engine for the newconfig can load the binary
format of the "files" file, so all features supported in the new-bus
also can be supported by the newconfig.

Furthermore,
in the new-bus, the user have to specify the following command:
	kldload if_de_cardbus.ko
In the newconfig, the user have to specify the following command:
	dconf "de* at cardbus?"
to supply cardbus-specifc attachment kernel module.

As this shows, the newconfig doesn't need more information than the
new-bus. The difference is the newconfig always uses device name, and
the new-bus sometimes uses device name (e.g. for configuration hint)
but sometimes uses module name (to load the module).

Actually the newconfig is better in this point, because it always uses
specification (device name "de* at cardbus?") instead of the
implementation (module name "if_de_cardbus.ko"). 
Thus, when the user upgrades FreeBSD's version, he doesn't have to
change the configration information, because the configuration
information doesn't depend on specific implementation.

Note that this device name to module name mapping information is
needed anyway, to support on-demand device loading.

The newconfig is well considered than the new-bus.

> 3: we want to better support seperate compilation where 3rd parties can build
>    modules that can interface cleanly even if the new-bus configuration
>    interface has changed to have more object methods.

Of course this can be done in the newconfig, too.
The newconfig doesn't have any limitation in this area.

> Now, let me see if I understand your counter-claims:
	:
> Is that a fair summary of your position? 

Yes.

> To save some time, I'd like to suggest some responses to the points
> above..

> 1: Yes, config.new has a far better implementation than our old config(8)
>    for this sort of thing.  Our plan all along has been to discard
>    config(8) as soon as possible and replace it with a tool specifically
>    designed to be kld and loader(8) aware, using tables loaded at boot time
>    from a text file for initial hints for device id -> driver matches.

But apparently you cannot discard all static configuration.
And as far as the static configurated part remains, the newconfig is
better, because it can handle both static and dynamic configuration in
same way.

> 2: This is a matter of preference..  It ends up in the same place.  While
>    the newconfig/config.new method probably saves some repetition, it doesn't
>    have the same flexibility that the in-source free-form method has.

That's wrong.
All flexibility which the new-bus currently has can be supported by
the newconfig. And, the newconfig can supports static configuration,
but the new-bus cann't. Thus the newconfig is more flexible than the
new-bus.

> 3: I do not doubt that you could extend the kernel components to be dynamic,
>    but I suspect it would be a lot of work.

The reason why dynamic configuration for the newconfig requires more
work than the new-bus is because it has some important features which
the new-bus doesn't have.

> 4: Yes, it would help.  But there are a lot of other differences that would
>    still make a fairly major task of decent driver portability.  I strongly
>    suspect that the changes required to make decent dynamic configurability
>    and load/unload etc support that we would require would damage this very
>    same portability goal.

This is completely misunderstanding.
The dynamic configuration for the newconfig doesn't require
the changes to the drivers. Only configuration mechanism have to be
changed, thus, all part of the drivers can be shared between all *BSDs.
And, of course the changes to the configuration mechanism also will be
able to be shared between all *BSDs.

> I do not believe that newconfig could be enhanced as effectively, at
> least not without damaging driver portabilty.

This is misunderstanding as said above.

> I expect that you are bitterly disappointed, and for that I sincerely
> apologize.  I do hope that your efforts on enhancing the drivers for
> better configuration so far can be salvaged and I hope very much that you
> can and will at some time consider working with us in the future..

Probably some of us will go to the new-bus, and some of us will stick
to the newconfig at least until the dynamic configuration for the
newconfig is completed.
Please don't blame the members who stick to the newconfig, because
what they'd like to do is proof of the concept.
- --
soda

------------------------------

Return-Path: imp@harmony.village.org
Received: from sramhc.sra.co.jp (root@sramhc [133.137.20.31]) by srapc342.sra.co.jp (8.8.8/3.4W-sra) with ESMTP id PAA17737 for <soda@srapc342.sra.co.jp>; Sat, 24 Apr 1999 15:45:15 +0900 (JST)
Received: from sranha.sra.co.jp (sranha.sra.co.jp [133.137.8.8])
	by sramhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id PAA11886;
	Sat, 24 Apr 1999 15:45:13 +0900 (JST)
Received: from sraigw.sra.co.jp (sraigw-hub [133.137.8.14])
	by sranha.sra.co.jp (8.8.7/3.6Wbeta7-sranha) with ESMTP id PAA23371;
	Sat, 24 Apr 1999 15:45:11 +0900 (JST)
Received: from rover.village.org (rover.village.org [204.144.255.49])
	by sraigw.sra.co.jp (8.8.7/3.6Wbeta7-sraigw) with ESMTP id PAA20926;
	Sat, 24 Apr 1999 15:45:10 +0900 (JST)
Received: from harmony.village.org (harmony.village.org [10.0.0.6])
	by rover.village.org (8.9.3/8.9.3) with ESMTP id AAA75020;
	Sat, 24 Apr 1999 00:44:40 -0600 (MDT)
	(envelope-from imp@harmony.village.org)
Received: from harmony.village.org (localhost.village.org [127.0.0.1]) by harmony.village.org (8.9.3/8.8.3) with ESMTP id AAA10549; Sat, 24 Apr 1999 00:44:48 -0600 (MDT)
Message-Id: <199904240644.AAA10549@harmony.village.org>
To: Noriyuki Soda <soda@sra.co.jp>
Subject: Re: new-bus/newconfig 
Cc: Peter Wemm <peter@netplex.com.au>, Atsushi Furuta <furuta@sra.co.jp>,
        Satoshi Asami <asami@freebsd.org>
In-reply-to: Your message of "Fri, 23 Apr 1999 04:22:32 +0900."
		<199904221922.EAA05764@srapc342.sra.co.jp> 
References: <199904221922.EAA05764@srapc342.sra.co.jp>  <199904170849.RAA07907@srapc342.sra.co.jp> <19990417105052.49C511F5D@spinner.netplex.com.au> 
Date: Sat, 24 Apr 1999 00:44:48 -0600
From: Warner Losh <imp@harmony.village.org>

In message <199904221922.EAA05764@srapc342.sra.co.jp> Noriyuki Soda writes:
: I'm sorry for quite late response.

No problem for me....

: But the newconfig mailing list itself is public mailing list, anyone
: can subscribe it, and the archive of it is accessable/searchable from
: WWW. (see http://www.jp.freebsd.org/newconfig/)
: 
: How do I report this discussion to the newconfig mailing list members ?
: Perhaps, is it OK to send this to each member of mailing list ?
: (to prevent this discussion from being included in archive)

Personally, and you'd have to ask Peter for this, I think that it
would be OK to post to the newconfig list, with Peter's disclaimers
about how he was speaking....

: > 1: kernel configuration is presently too hard.  Too many people have enough
: >    trouble with a kernel config file at all.  It needs to be made easier
: >    and not more complicated.  To my mind (and many others) the config.new
: >    file format and syntax requirements are excessive.
: 
: The newconfig group once already discussed this issue.
: In the newconfig, the utility like Linux's "make menuconfig" can be
: made and can be able to used for both static and dynamic configuration.
: It must work better than the Linux "make menuconfig", because it can
: retrieve the bus/device hierarchy from "files" file.
: In the new-bus, such utility cannot be made for neither static nor
: dynamic configuration.

This is a fair critism, as far as it goes.  new-bus is projected to
take care of this via hints.  Many buses are self identifying, so for
those busses it makes sense to have things be as automatic as
possible.  In the case of PCI, for example, there exist a many to one
mapping of a device ID to a driver.  The user doesn't need to know
about this, and with one exception (hardwiring) the user doesn't care.

For ISA-like buses, you'd need to still have a table of hints to
figure out what is on the bus.

The direction that I've heard people would like to see new-bus head is
that the kerenl will have just enough static compiled into it to mount
its / file system.  For most users, this will be the disk drivers, the
console and UFS, although for a netboot this might be NFS and a
network driver.  Once the system has been loaded and the root file
system has been mounted, then a directory can be scanned, with all the
devices in it loaded, probed, and unloaded if the probe fails.  This
is much like how OS/2 did its device drivers.  To add support for a
new device, you just copy its driver over.  Most of the mechanism to
do this isn't presently in the newbus implementation.  And most of
these impressions have come from lunch time conversations with Justin
Gibbs, whom I work with.

I've heard different ways of actually doing this.  One is to load all
modules in a directory into the kernel, and have the unused ones
unload sometime after it has been determined they are unused and
reclaim the memory.

: Furthermore,
: in the new-bus, the user have to specify the following command:
: 	kldload if_de_cardbus.ko
: In the newconfig, the user have to specify the following command:
: 	dconf "de* at cardbus?"
: to supply cardbus-specifc attachment kernel module.

While the kldload is currently required in what is implemented in the
repository today, in the future it may not be required at all.  The
above mechanism would take care of loading things that make sense to
be loaded.

: As this shows, the newconfig doesn't need more information than the
: new-bus. The difference is the newconfig always uses device name, and
: the new-bus sometimes uses device name (e.g. for configuration hint)
: but sometimes uses module name (to load the module).

This is a fair point as new-bus stands today.

: Note that this device name to module name mapping information is
: needed anyway, to support on-demand device loading.

I'm not so sure that this is the case (to have the mapping from device
name -> module).  All I think that is required is the other way
(module name -> device(s)).

: The newconfig is well considered than the new-bus.

I don't know what the plans are for complicated device
interdependencies are, so newconfig would be better at solving that
problem because it encodes the entire tree the user is interested in.

: Of course this can be done in the newconfig, too.
: The newconfig doesn't have any limitation in this area.
: 
: > Now, let me see if I understand your counter-claims:
: 	:
: > Is that a fair summary of your position? 
: 
: Yes.
: 
: > To save some time, I'd like to suggest some responses to the points
: > above..
: 
: > 1: Yes, config.new has a far better implementation than our old config(8)
: >    for this sort of thing.  Our plan all along has been to discard
: >    config(8) as soon as possible and replace it with a tool specifically
: >    designed to be kld and loader(8) aware, using tables loaded at boot time
: >    from a text file for initial hints for device id -> driver matches.
: 
: But apparently you cannot discard all static configuration.
: And as far as the static configurated part remains, the newconfig is
: better, because it can handle both static and dynamic configuration in
: same way.

This is a fair argument in favor of newconfig, as new-bus stands
today.  However, talking with folks about the direction for new-bus is
that people would eventually like to see config go away completely.
You'd build the "core" part of the kernel, and everything else would
be a loadable module, ala solaris.  There would also be a utility to
merge modules together (maybe ld does sufficiently well already) for
those that want or need to have a more restricted set of drivers.

: > 2: This is a matter of preference..  It ends up in the same place.  While
: >    the newconfig/config.new method probably saves some repetition, it doesn't
: >    have the same flexibility that the in-source free-form method has.
: 
: That's wrong.
: All flexibility which the new-bus currently has can be supported by
: the newconfig. And, the newconfig can supports static configuration,
: but the new-bus cann't. Thus the newconfig is more flexible than the
: new-bus.

As things stand now, this is a fair complaint about newbus.  IF newbus
goes in the direction it is headed, the need to have static
configuration will diminish over time.

: > 4: Yes, it would help.  But there are a lot of other differences that would
: >    still make a fairly major task of decent driver portability.  I strongly
: >    suspect that the changes required to make decent dynamic configurability
: >    and load/unload etc support that we would require would damage this very
: >    same portability goal.
: 
: This is completely misunderstanding.
: The dynamic configuration for the newconfig doesn't require
: the changes to the drivers. Only configuration mechanism have to be
: changed, thus, all part of the drivers can be shared between all *BSDs.
: And, of course the changes to the configuration mechanism also will be
: able to be shared between all *BSDs.

>From looking at the shared drivers between NetBSD and FreeBSD today,
this is mostly true, except where it isn't.  That is to say one
couldn't have a SCSI driver easily shared between NetBSD and FreeBSD
today (but one could, ala the esp driver, do it with enough macro
pain).  Many other classes of drivers, say the networking code, do
have enough in common that making configuration common as well would
almost render them portable.

: > I expect that you are bitterly disappointed, and for that I sincerely
: > apologize.  I do hope that your efforts on enhancing the drivers for
: > better configuration so far can be salvaged and I hope very much that you
: > can and will at some time consider working with us in the future..
: 
: Probably some of us will go to the new-bus, and some of us will stick
: to the newconfig at least until the dynamic configuration for the
: newconfig is completed.
: Please don't blame the members who stick to the newconfig, because
: what they'd like to do is proof of the concept.

>From lurking at the edges of this debate, I think that the newbus and
newconfig represent different world views to solve the same problem.
As it stands today, newconfig is better for some things, and worse for
others.  The same is true for newbus.  Neither one is a static,
completed entity, but are both growing and dynamic projects which
makes it difficult to judge the differnces and similarities.  And at
some level it boils down to ones ideology on these things.  I'm not
sure how to bridge that gap.

Warner

------------------------------

Return-Path: soda@sra.co.jp
Received: from sramhc.sra.co.jp (root@sramhc [133.137.20.31]) by srapc133.sra.co.jp (8.8.5/3.4W-sra) with ESMTP id WAA18645 for <soda@srapc133.sra.co.jp>; Sun, 25 Apr 1999 22:18:05 +0900 (JST)
Received: from sranhc.sra.co.jp (sranhc.sra.co.jp [133.137.20.3])
	by sramhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id WAA17602;
	Sun, 25 Apr 1999 22:18:02 +0900 (JST)
Received: from srapc133.sra.co.jp (srapc133 [133.137.21.8])
	by sranhc.sra.co.jp (8.8.7/3.6Wbeta7-srambox) with ESMTP id WAA25602;
	Sun, 25 Apr 1999 22:18:02 +0900 (JST)
Received: (from soda@localhost) by srapc133.sra.co.jp (8.8.5/3.4W-sra) id WAA18641; Sun, 25 Apr 1999 22:18:02 +0900 (JST)
Date: Sun, 25 Apr 1999 22:18:02 +0900 (JST)
Message-Id: <199904251318.WAA18641@srapc133.sra.co.jp>
From: Noriyuki Soda <soda@sra.co.jp>
To: Warner Losh <imp@harmony.village.org>, Peter Wemm <peter@netplex.com.au>
Cc: Noriyuki Soda <soda@sra.co.jp>, Atsushi Furuta <furuta@sra.co.jp>,
        Satoshi Asami <asami@freebsd.org>
Subject: Re: new-bus/newconfig 
In-Reply-To: <199904240644.AAA10549@harmony.village.org>
References: <199904221922.EAA05764@srapc342.sra.co.jp>
	<199904170849.RAA07907@srapc342.sra.co.jp>
	<19990417105052.49C511F5D@spinner.netplex.com.au>
	<199904240644.AAA10549@harmony.village.org>

> : But the newconfig mailing list itself is public mailing list, anyone
> : can subscribe it, and the archive of it is accessable/searchable from
> : WWW. (see http://www.jp.freebsd.org/newconfig/)
> : 
> : How do I report this discussion to the newconfig mailing list members ?
> : Perhaps, is it OK to send this to each member of mailing list ?
> : (to prevent this discussion from being included in archive)
> 
> Personally, and you'd have to ask Peter for this, I think that it
> would be OK to post to the newconfig list, with Peter's disclaimers
> about how he was speaking....

Peter, do you think so, too?
May I post this discussion to newconfig mailing-list, if the following
disclaimer is attached ?

	On these mails, Peter was discussing as a developer, not as an
	official core discussion.  He wrote with the understanding
	that it was between newconfig group and Peter.

> The direction that I've heard people would like to see new-bus head
> is that the kerenl will have just enough static compiled into it to
> mount its / file system.  For most users, this will be the disk
> drivers, the console and UFS, although for a netboot this might be
> NFS and a network driver.

As long as static compiled modules remains, the new-bus doesn't work
well with these static modules.  In the newconfig, the user is free to
choose static or dynamic. The new-bus isn't.

> Once the system has been loaded and the root file system has been
> mounted, then a directory can be scanned, with all the devices in it
> loaded, probed, and unloaded if the probe fails.  This is much like
> how OS/2 did its device drivers.  To add support for a new device,
> you just copy its driver over.

If this is really required feature (*1), this can be easily
implemented in the newconfig, too. Because the newconfig already holds
all necessary information to achieve this.

(*1)	Though I don't think so. It actually increases maintenance cost
	of OS version up.

> I've heard different ways of actually doing this.  One is to load all
> modules in a directory into the kernel, and have the unused ones
> unload sometime after it has been determined they are unused and
> reclaim the memory.

This also can be done in the newconfig.

> : Note that this device name to module name mapping information is
> : needed anyway, to support on-demand device loading.
> 
> I'm not so sure that this is the case (to have the mapping from device
> name -> module).  All I think that is required is the other way
> (module name -> device(s)).

Then, on-demand device module loading never can be achieved in the
new-bus.

> This is a fair argument in favor of newconfig, as new-bus stands
> today.  However, talking with folks about the direction for new-bus is
> that people would eventually like to see config go away completely.
> You'd build the "core" part of the kernel, and everything else would
> be a loadable module, ala solaris.  There would also be a utility to
> merge modules together (maybe ld does sufficiently well already) for
> those that want or need to have a more restricted set of drivers.

In some applications, dynamic configuration is merely redundant.
The new-bus is not suitable for these applications.

Even in dynamic configration, the new-bus lacks some important
features which the newconfig has. So, the new-bus eventually have to
reinvent these features.

I cannot find any point where the new-bus is better than newconfig in
it's design. Why do the new-bus people have to reinvent whole
configration scheme?  Isn't the new-bus NIH?
- --
soda

------------------------------

End of digest
***********************************************
--
soda@sra.co.jp		Software Research Associates, Inc., Japan
(Noriyuki Soda)			Advanced Technology Group.
