Shaking off the C shackle

Software
1

Greetings Gentlemen,

I have a little free time on my hands again, and I was looking at my retro bookshelf. .

On my shelf I have Tannenbaum’s OS Design and Implementation (which I’ve studied), Comer’s XINUX book (which I’ve skimmed), and some familiar suporting literature, like Bach’s “Design of the Unix Operating System” and Plauger’s “The Standard C Library.”

I was pondering how this approach to OS design depends on preexisting support for C (in the form of ACK and libraries for MINIX, e.g.) and results in operating systems that are, in a kind of spiritual way, just big complicated C interpreters.

In Plauger, for example, when you trace a call down to the bare hardware, you eventually get to a point - the point that makes everything work or not work - where he says, more or less, “you have to write this part in assembly language. Either copy one that works from an existing system, or do it from scratch: good luck!” And that’s it. Tannenbaum includes the assembly code, with a bit of commentary, but doesn’t tell you how to do it yourself. Comer dodges the whole thing by using BIOS calls for everything (at least in the PC version I have).

I mentioned in my older post how, while appreciating its importance and utility for real world systems, I find a lot of abstraction really irritating when I’m trying to learn how things work. It occurred to me that studying OS design based on C is probably a bad idea in the first place, since the whole “black box” virtual machine is the entire point, and one of the reasons for C’s wide historical success.

So, suppose it’s the the late 70s / early 80s and I want to learn how to write an OS for my CoCo. What textbook do I buy? Is there such a textbook? Did Gary Kildall invent all the techniques he used to write CP/M (if so: WOW!) or was there an OS:DI equivalent for 8-bit and 16-bit systems?

3 Likes
2

I suppose the way I’d think of it is that C is an adequately low-level language to do almost all the work of the OS, but also an adequately high-level language to be productive and portable. It does seem that C can’t quite manage the full 100% - perhaps because it has no way to emit some small number of crucial instructions which are needed, other than inline assembly code.

I wonder if you would read a textbook, in this case. Perhaps what you’d draw on is practical knowledge, of using and perhaps reverse-engineering - or at least, deeply understanding - some existing OS on some existing hardware. Perhaps a minicomputer.

Famously, Linus Torvalds started by writing a task switcher. Which I think illustrates a very very important question: what do you expect your OS to do? It would be natural, I think, to start with the absolute minimum, and build up, but not build up any further than you felt the need.

One might, then, ask this: what OS functions does CP/M offer? Or the very first Unix? Or the very first Linux? I suspect ‘glorified program loader’ would cover a major part of it - and I don’t mean that in a demeaning way. If I can load and run a program, if that program can read and write files, it if can take input from the keyboard and output to the screen, that’s an operating system. For sure it’s handy to have things like multitasking, interprocess communication, memory protection, multiple users, abstraction layers for input and output and filing systems - but none of that is essential, just very handy.

It’s a historical question - and an interesting one - as to how Kildall got the knowledge he needed. I don’t know the answer.

3 Likes
3

I was kind of afraid that might be the case.

I’m not sure if the rest of your post was an answer or a question - I can read it both ways, and I agree with it both ways (if it’s possible to agree with a question!). For me, personally, I would be satisfied if I could, at least as a starting point, create something with about the same baseline functionality as the BASIC ROM in my CoCo. It has a command shell, can do character I/O, basic file management (now that I’ve upgraded it with a CF floppy emulator cartridge), and can perform monitoring functions, like reading / writing memory, and executing machine code. (Actually it is more robust than that; I gather as BASICS go, the CoCo implementation is a decent one.)

Thinking along the lines you suggest, I suppose that writing a loader isn’t such a big deal for someone who writes assemblers and compilers, and writing a command shell is really a higher-level problem (I actually wrote one in SCHEME a long time ago).

Here’s a refinement: instead of “what came before C / UNIX,” what about the PC BIOS and DOS? Early DOS was often little more than a (sometimes not so) convenient wrapper for BIOS calls. What did those IBM guys study that let them create such a robust system that after 40 years we’re only just now discontinuing the use of some parts of it?

Putting that in personal terms, maybe my project should be “I’d like to write a BIOS for my CoCo2 in 6809 assembly language, and then use that to make a monitor program that offers basic OS functions like command interpretation, program loading, etc.” Task schedulers, e.g., and such could easily be added later on if they were useful or interesting.

Although I don’t have “Bootstrapping Forth,” I am slowly reading “Threaded Interpretive Languages” by Loeliger, which seems at least adjacent to this topic; sort of solving the same problem but from a different angle.

Thanks for your reply Ed! Very thought provoking.

3 Likes
4

Quick addendum, it seems Gary Kildall had a strong academic background in computer science, and had built himself a computer in a briefcase:

From the transcript:

… the first thing I heard of that Gary did that was really brought to my attention was he he’d invented a programming language called PLM and implemented it for the Intel microprocessors to prove that that the 8088 was the 8080 I’m sorry was a real computer and not a controller for for microwave ovens but it was a real computer and he went off and wrote a programming language that ran on microcomputers know we can say well well of course that’s no big deal but at the time it was pretty big deal he invented this language and then to show that the language was useful he wrote CPM that’s what really actually happened he created this operating system and and built it around this Intel microprocessor to show what could be done with microprocessors and in 1975 when he was doing this that was pretty revolutionary.

2 Likes
5

C was very much a thing in the late '70’s. As was Unix. But making C run on an 8-bit micro (6809 in the CoCo) was thought to be virtually impossible, so what else?

Cross the Atlantic (assuming you may be “left pondian” in the first place) to Edinburgh University and there was another multi-user OS, EMAS. The Edinburgh Multi Access System. Written in another Algol-like language; Imp77.

What else? Well, before C there was (very briefly) B and before that BCPL.

And one notable OS written in BCPL is OS6 - written in BCPL running on modest hardware at that time.

Mentioned here: OS6 - a portable operating system (1972) and because it was done in academia then all the sources, etc. were available - if you knew where to look.

So you want to write an OS in the 70’s - then I suspect it would depend a lot on your background (academia, industry?) and possibly also what continent you were on… (No Internet then!) DEC, HP, Prime, Data General, Interdata and many others in the “mini” market all had their own operating systems.

One notable observation (to me)… is what pre-dated Unix on DEC systems? OS8 on the PDP/8 and various operating systems on the PDP11… And there is where I feel Kildall may have gotten some of the inspiration for CP/M. (If nothing else, some of the commands are the same - see for example the PIP program described here: https://en.wikipedia.org/wiki/Peripheral_Interchange_Program )

Also note that some of CP/M was written in PL/M.

Today - I find it hard to get away from Unix - I’m writing my own little retro style OS in BCPL, running on retro-style hardware (W65C816) but I just can’t get away from making it look like Unix - to the point that it’s multi-tasking (although only single user, but, like CP/M, it could be many users, but only one at a time - is there a word for that?) and has a Unix-like filing system and command utilities. I also feel that with a nod to your previous post it might well be at the limit of complete understanding from one persons point of view… However while I didn’t write the BCPL compiler, I did write (and re-write/modify) almost all the system libraries

And a modern take on an OS for the old 8-bitters is Fuzix - and guess (from it’s name!) what it might resemble…

-Gordon

3 Likes
6

That is a great video; I watched the whole thing. I’ve read a fair bit about Gary Kildall in wikipedia articles and various retrocomputing sites, but it’s cool to see interviews with so many people who personally knew him.

1 Like
7

For me, this is the big difference between the (typical) minicomputer OS and the (typical) microcomputer OS in the 70s and early 80s. The microcomputer OS was a program loader and a bag of routines that, if you squint, kind of looked like device drivers; this includes CP/M, Apple DOS and ProDOS, Tandy DOS for the 6809, and even MS-DOS/PC-DOS. On the other hand, the minicomputer OS had a task scheduler, IPC, and memory protection – although maybe not multiple users or a substantially more complex filesystem.

The systems that cross this gap are very interesting to me; RT-11SJ and OS-9 Level 2, for example, are a minicomputer OS that is little more than CP/M and a microcomputer OS that’s almost Unix. (Some DEC and Motorola fans are going to come string me up, now.) For the topic at hand, they’re particularly interesting because they were often paired with a version/implementation on the other side of that gap – OS-9 Level 1 had no memory protection, for example, and RT-11XM had priority scheduling and multi-TTY capability.

I agree that minicomputers are probably a good place to look for building something from the ground up, and the reason is that their native device functionality is often nearly as complete as the BIOS routines mentioned when discussing PC-XINU. There is, for example, an older version of the XINU book that targets the LSI-11 that might be appropriate; it does still require a C compiler, but the deepest interaction with the hardware is PDP-11 assembly speaking to hardware devices. Microcomputers are often more difficult, and intended to be interfaced using a BIOS or sophisticated ROM routines provided by the vendor; certainly this was the case for CP/M machines, the Apple ][, C64, etc. It’s not that you couldn’t interface the hardware directly, but it often wasn’t done. On the other hand, you can find piles of books like J.W. Cooper’s The Minicomputer in the Laboratory, which starts from the premise that you have a PDP-11 and an assembler and some work to do, and walks through bespoke assembly language implementations of interesting computations on raw hardware. (It does also discuss RT-11 and, I think, RSX-11.)

2 Likes
8

This reminds me of Lyons’ Commentary. I’ll save these for a closer look!

9

I’ve been looking on and off for a good condition but affordable used copy of that book for a while now. Thanks to your reminder I just checked again, and found one!

This is exactly the kind of thing I was hoping to find! I guess I don’t need more than that one (which I just found on BWB for $8), but I like this idea of piles. Where should I look to find more like that?

2 Likes
10

This sounds highly reasonable to me! It’s a bonus if you can somehow assume or inherit a filing system… although if you only have cassette, rather than floppy, it’s so sequential that you barely need more than a length and a payload. (A monitor program which can load srecords is quite handy. In fact these days, where we can assume a more capable computer nearby, talking over a serial port, both for commands and data, can be enough to feel like you’re getting somewhere.)

2 Likes
11

Uhh … I’m not sure that all of these have that kind of material, but most of them do, and these are just the ones that I have in my personal collection:

  • The aforementioned The Minicomputer in the Laboratory by J.W. Cooper
  • Minicomputer Systems: Organization and Programming by Richard Eckhouse
  • Machine and Assembly Language Programming of the PDP-11 by Arthur Gill
  • The Digital Way: Macro-II Assembly Language Programming by James Peters

If you periodically troll the usual sites for “PDP-11” or other minicomputer architectures, you’ll find many more! Some of them are not very good (I don’t particularly like The Digital Way, above, for example, although maybe others do), and it’s hard to know before you buy, so look for deals. :wink:

There are also a lot of books on just generically assembly language programming or interfacing that talk about some of this stuff, but aren’t necessarily targeted toward bare metal development; for example, Programming 16-bit Machines: The PDP-11, 8086, and M68000 by William H. Jermann or Minicomputer and Microprocessor Interfacing by John Cluley (again, to name a couple in my collection; my collection is mostly driven by “I found it and it was cheap”, so there’s a wide world outside of this!).

2 Likes
12

On a somewhat different tack, you might enjoy something like Nicklaus Wirth’s Oberon system and associated books (http://www.projectoberon.com/). It’s basically the same thing except in Oberon, which is a follow-on language to Modula-2.

There are also some very interesting Forth systems out there from the FIG-Forth group, several of which (including the PDP-11 version and I believe some iterations of the the CP/M version) are capable of running either under an operating system (RT-11 for the PDP-11 version, for example, or CP/M) or on bare metal with very specific devices and hardware (RX01 drives for the PDP-11 version, again).

There are also some systems like Jonesforth or Planckforth (See also Bootstrapping Forth from a 1K ELF), which target the Linux syscall interface in much the same way that PC-XINU targets the PC BIOS, or SectorLISP which targets the PC BIOS but provides a LISP environment directly (but might be a bit less of a “usable” system than Jonesforth or Planckforth). All of these are more exercises in minimalism than user environments like an OS, of course.

1 Like
13

A compiler of any kind requires about 16K words of code and 8K words of data
and a operating system for file I/O. POOF 64KB of memory is now used up
and you wonder how things fit on a 8 bit micro. C is trickey because it has the long
data size and 64 bit floating point from the PDP 11, with kernal and user program/data
spaces. Memory was the limiting factor, not what cpu is used. Mind you 32K of memory
at one time could be found on big main frames, like the IBM 360, thus you had a complex
languge like PL/1 with multi-pass compilation.
C was a 3 pass compiler, macro expansion, c to assembler,assembler to object code.
A good compromize for the time, but requires real file storage. Unix being a multi user
operating system,also requires fast I/O for swap file space and other virtual memory
functions, big money in the 70’s, but alot less money than a main frame.
C is with us because a language moved from punched cards to stream I/O
and that was biggest feature at the time.
OS9 level II is what you want for the 6809. http://matchboxcoco.com/ looks
to be a modern version.

14

The BIOS is what these days are called “hardware abstraction layer” (HAL). I am not aware of an OS before Gary’s CP/M that had such a good separation between the HAL and the OS itself. The IBM guys just copied it (including the very name, “BIOS”).

The BIOS only worked in the x86 real mode, so could be used by OSes such as DOS, CP/M or UCSD but any protected mode OS (such as QNX, later Windows, Linux and many others) had to include their own complete replacements for the BIOS. Much later the BIOS evolved to be able to be used from the protected mode, but it got mostly replaced by UEFI (“Unified Extesible Firmware Interface”) before that feature became widely available, if I remember correctly.

Speaking of extensible, that was a limitation of the original PC (IBM 5150) BIOS. Adding a hard disk to that machine was very awkward, for example. For the PC XT (5160) they improved it so an expansion card (the MFM hard disk controller, for example) could include a ROM with routines that would get inserted into the chain executed when Interrupt 13 was called. Now your hard disk could be treated exactly like a big floppy as soon as the machine was turned on, making it possible to boot directly from it.

So you see, this is a learning experience. IBM didn’t get it right the first time so nobody else should feel that they have to.

1 Like
15

Robust system is not what comes to mind for DOS. FLOPPY and VIDEO/KEYBOARD
was it. Real serial I/O no faster tha 1200 baud. Snow with every srcreen refesh.
4 colors for your TV display.
DOS was the best of all the other OS’s out there that I know of, as every thing else was
limited to 64Kb programs, or well over priced.
Ben.

16

Interesting notes on the evolution of the PC BIOS there, thanks!

Portability and extensibility are good engineering and business goals, most of the time, but a good simplification for a hobby system - especially one’s own first effort - would (I think) be to ignore those possible goals, and just try to build something which works, and is moderately maintainable.

Of course I might be naive in thinking this!

17

I’ve started a new topic for learning how to do things on 8-bit micros by studying minicomputers and mainframes. That seemed like it was enough different from where we started to deserve its own thread.

@oldben, just to make sure I haven’t been confusing, I do know about things like OS9; it’s not that I want an existing system to use, or that I think I can improve on what others have done. It’s just that I want to be able to do something in the first place. In service of a ground-up understanding I want every part of the system to work because I made that part work myself. Even if it doesn’t work very efficiently, or can’t be ported easily, etc.

It’s a part of my character that I’m pretty confident I can read my way out of any problem; if I just find the right books I can get what I need to know.

If I study only the source code of some fairly large project - like OS9, say - I could probably eventually figure out how it works. I think this would take a long time, though, and I’m afraid I would miss the larger relational concepts of how and why it got to be that way.

I have my CoCo 2 and a cartridge that lets me use my own code instead of the on-board ROM. At some point I have to stop reading (or maybe before I stop reading!) and start putting my own bytes in there. Right now I’m not sure what those first bytes should be. @EdS mentioned that Linus started by writing a task switcher; I might not need one of those. I’m hoping that some books like the ones @elb recommended will help me devise a clearer design vision. I feel like “Write a HAL for the CoCo2 in 6809” is a good goal; now I have to figure out what that looks like and what the first steps are for building one!

1 Like
18

For my home brew 20 bit computer, I have a simple front panel, with boot strap loader
and math utiity/ debug routines in rom. Disk I/O is block read/write and bootstrap.
At the moment I am adding BCD math so I can have floating point numbers.
This setup lets me play with software, and not have to program my roms that often.
Ben.

19

I’d say the same about CP/M.

Didn’t the x86 have the 64k barrier too?

So CP/M-(Z)80-v3 with banked memory would not have been technically worse than DOS with it’s 64k pages.

CP/M on chips with linear RAM bigger than 64k existed too.

And was the Atari-ST’s TOS a DOS rewrite from the original CP/M inventors or was it a modern CP/M-68k with directories?

The borders are unclear?

DOS always looked like a CP/M rewrite even before I heard about the background story. It even had no directories in its 1st version and some of the earliest DOS programs were patched CP/M-86 products.

20

Not in the same way. While it is true that each segment was limited to 64 kB offsets, there were multiple segments (code, data, stack, etc.) that were automatically used by the processor at appropriate times. At the very least, it was trivial to use a 64 kB program with 64 kB of heap and up to 64 kB of program stack without messing with banking, overlays, etc. The 8080/Z80 (where CP/M ran), on the other hand, had access to at most 64 kB of RAM at one time in total.