August 2023 – Bitser

Memories, Dreams and Refractions: Network Products Part I

In 1981 Business Application Systems was making a version of the BASPort portable operating system (private labeled VOS) for SCI Systems, a corporation based in Huntsville Alabama. That spring BAS agreed to sell the portion of BAS developing VOS to SCI.

Flash back to the 1960s when the integration of components for the Saturn V Instrument Unit was taking place in an IBM facility in Huntsville Alabama. SCI, then called Space Craft Incorporated, had created two of the boxes to be bolted into the IU. My father was the engineer responsible for determining that those boxes worked properly (backed up at a less technical level by NASA engineers). In the course of working with SCI employees my dad got a good feel for the SCI corporate culture as strongly formed by the asshole who ran it by the name of Olin King. Dad’s reports left me with a crystal clear judgement about whether I’d jump off a bridge or work for SCI.

So when news of SCI acquiring us was sprung I started shopping for a new job. By what I now consider to be one of the biggest coincidences that I’ve never experienced, a month or two after the acquisition Bob Nichols and Steve Schleimer approached me about joining their new startup Network Products that was chartered to make data communications equipment. I jumped at it. Working with Steve again was going to be simply sublime, as he’d been my mentor at Data General (he authored the virtual machine of the commercial language system I worked on). And Bob and I had got along pretty well at BAS (Bob wrote the commercial system’s compiler). Steve had left Data General where he was a software architect and developer of Data General’s Fountainhead project while Bob had already left BAS. Also joining us were Steve Hafele, a hardware design engineer also from Fountainhead and Steve Chewning, an ex-HP hardware design engineer.

Babymux

To be continued.

Steve Goldman 1952-2008

Steve Goldman front and center

Steve Goldman grew up in Pittsburgh, moved with his family to south Florida, went to school at Carnegie Mellon (BS in Electrical Engineering), married his sweetheart Gretchen (BA in Fine Arts, also from CMU) and worked for Motorola doing hardware work. He took a job with Texas Gulf in Raleigh, NC doing software work related to mineral exploration. He lived with Gretchen in neighboring Cary, NC where they had cats instead of kids.

After transferring with Data General from Westboro, MA to the same area of NC I joined a startup called Business Application Systems that spun out of Data General in January of 1978. No sooner had we incorporated than DG sued frivolously and we were in the courts for a year or two before we could start work on a portable operating system product called BASPort. While in litigation we all had to find work of any kind we could to keep afloat. The company immediately became an OEM of Texas Instruments series 990 minicomputers. The terminals for these minicomputers were proprietary and cost a few thousand dollars each. We brainstormed and realized we could write a software driver that would make a much cheaper serial terminal emulate the TI terminal. I wrote the driver and we sold it for $500 a copy. Then we got prospects for customers that wanted to put a group of these cheap terminals and printers in location A while their TI minicomputer was in location B. That called for a statistical multiplexer that presented a set of serial ports on each end of a leased digital phone line, transparently moving the data as if the terminals and printers were directly connected to the minicomputer.

When I had the terminal emulator up and running we hired Steve Goldman and he joined me to take over the terminal emulator while I started on the statmux. We shared that code development and placed several with customers at several thousand dollars a throw.

Steve was getting a masters degree in computer science from UNC/Chapel Hill and over the course of the next few years caught developed both expertise and focus. Steve fell in love with compiler construction and stuck with that from 1980 onwards (28 years total). After the litigation we kept the datacomm plates spinning and turned attention to BASPort. Steve developed compilers that targeted the BASPort virtual machine.

This went on until 1981 when I was offered a job in the startup Network Products writing firmware for standalone and modular datacomm products. But we were on the first floor the the building where Steve was on the third floor and we often had meals together and hung out away from work. So instead of being a few feet away Steve was two floors away for a while. Then in 1984 Steve and I both got jobs at Encore Computer. The two of us worked together as the third generation language development part of the company and we beavered away on Pascal, C, Fortran and C++ compiler development for the company’s symmetric multiprocessor computer systems (see link above for more details).

By the early 90s Encore’s ability to compete selling general purpose computers was fizzling and they were left with a few niche areas like supporting flight simulators. But the company developed smart storage systems that provided IBM mainframe plug compatible storage at extremely competitive prices. Steve worked on the Unix kernel drivers that interacted with storage at at a low level.

In 1997 Sun Microsystems bought Encore to get its storage products. Steve continued kernel work, but this time in Sun’s Unix kernel. This went on for a year before he and I were forced to transfer to the Java Technology Group within the Solaris Software division of Sun. After a very tense few months spent evaluating competing Java virtual machines for management we began development of parts of the Java virtual machine that would become the next generation Java product on Solaris (and eventually on Linux). Steve soon migrated to the compiler group, working on the just in time compiler residing inside the Java VM. There were actually two compilers: one wimpy but fast and one slower but with state of the art optimizations for super efficient code. Steve migrated to the latter.

Over subsequent years Steve rose in the ranks to become one of the most senior and accomplished developers within the Java Engineering organization. He was very well respected. When a friend was interviewing to retain his job at SAP after SAP acquired Sybase where he worked, he was struggling to impress them at first. But the SAP staff interviewing him became excited and animated when the friend mentioned our names. “You worked with Steve Goldman?” they asked for confirmation, visibly awed. After he gave more details the friend was instantly blessed as worth retaining by this simple association. (SAP was a heavy user of Sun Java and had many ties to Sun engineering).

Steve was like Dirty Harry in many situations, dealing with the hardest problems that were otherwise shunned. He fixed one Java customer’s problem that was so intermittent it only surfaced once every few weeks and there were no hints at all about the source of the problem. Steve provided specially instrumented virtual machines month after month that snuck up on the failure and exposed it. This was a problem that was not affecting the other thirty million users of Sun Java: only the one that was coincidentally at a physical location in Raleigh, NC. A N Y other engineer could have been expected to declare the problem impossible to fix, management would look at the financial consequences of blowing the customer off vs spending many thousands of dollars of staff time and that would be that since the customer was not putting big bucks into Sun pockets. But that wasn’t how Steve operated, and the hundred or so hours that bug fix cost was something he simply got done. Finally, the root cause of the problem did not involve his internal compiler: it was an extremely subtle cache management issue involving a corner-corner-corner case that resulted in a stale line.

When we did compiler development at Encore we drove bug counts to zero and kept them there. An unfixed bug was simply intolerable. That attitude stuck with Steve even as the complexity of runtime problems inside Java as it executed an application were many orders of magnitude larger. I can’t stress enough how devilishly complex that software was.

When not working on software Steve built stuff, had challenging hobbies, read a lot and lavished loving attention on the cats. He got part way through building a large reflector telescope and completed an observatory with movable roof to house it. He built an aircraft hanger with almost no help. He repaired his cars for everything short of an engine overhaul. He and Gretchen designed a custom house and he was very involved in its construction. He and Gretchen raised two nieces as if they were their own children from ages 13 and 15 through university years, providing a very stable, loving environment that rescued the two girls and helped them lead happy and productive lives as adults. Steve helped others when they needed help, whenever it was needed. He got a pilot’s license and flew a Grumman Tiger and got well along with building a custom airplane in his hangar. He provided support to Gretchen as she pursued her creation of art and to other family members in their times of need.

In April of 1991 Steve was taking off with his hang glider when a technical problem disconnected the control bar. The glider pitched up, stalled and then fell down a few dozen feet into a concrete runway. He almost died. He spent weeks in a hospital and months in a hospital bed in his house. In October he walked the two miles from his house to our Encore office. He told me his shoe lace and come undone and he couldn’t bend over. I tied it as we cried together.

One year Steve decided he would get a “one” annual review rating. This was the highest rating possible. He walked on water while juggling chain saws. His creative output and engineering work were astonishing. Then he found out that management didn’t actually dispense ones on the basis of merit but via various maneuvers that I don’t want to discuss. Steve didn’t get his one. He broke down and cried as we hugged. He considered trying for a promotion to Distinguished Engineer until he found out the selection process was even more capricious. I had gotten a one rating the year before he tried and that hurt more because my work was pie in the sky stuff (an attempt to fix an archetecture bug in Java), so it was mostly perception with not much substance. But we both knew the score.

At the end of June, 2008 Steve and Gretchen, Katie and Little Gretchen, Steve’s mother, brothers and sisters were all on a cruise off the coast of Alaska. Steve had an aortic aneurysm and died a short time after feeling unwell. He was cremated in Seattle.

So Steve and I worked together for most of 30 years sitting within speaking distance. It’s been 15 years since he left us but my memories of him are still so sharp and close I can regularly ask myself “what would Steve do in this situation?” and see his expressions. He’s in my dreams.

Memories, Dreams and Refractions: Encore Computer Part I

Steve Goldman was still at SCI Systems (that had acquired part of Business Application Systems, the first startup we both worked at) while I was at a second startup called Network Products when we were offered positions at a new startup called Encore Computer in 1984. This was less than a year after its founding and very early in its process of acquiring and pulling together a set of organizations:

Hydra Computer Systems, a maker of symmetric multiprocessor (SMP) computer systems and developer of a multiprocessor version of BSD Unix in Natick, Mass.
Resolution Systems, a maker of somewhat smart computer monitors in Marlboro, Mass. Together with the Hydra guys, eventually laid off when the Marlboro plant was closed after most of the company resided in south Florida (after Encore swallowed the Gould Computer Systems Division whale using Japanese money).
Foundation Computers, a maker of a fourth generation application development language in Cary, NC. Sold to Unisys in 1985.
?? (forgot name) A network appliance maker, in particular the original creator of the Annex network terminal server, based in Marlboro, MA. Sold by Encore to Xylogics in 1986 as one of the all time bone head business moves of the 20th century.
?? (forgot name), a development shop porting Unix System V Release 4 (SVR4) based in San Diego, CA. These guys (originally Larry and ? but later maybe two other guys for four total). Laid off, discovering their layoff via an announcement by Ken Fisher at a trade show!
?? (forgot name), a group of consultants associated with Carnegie Mellon in Pittsburg, PA. These guys moved up to Marlboro or went on to other things as far as I could tell.

Although Steve Goldman and I were hired by Earl Gilmore¹, cofounder of Foundation, our charter from the beginning was development of the third generation computer languages for Encore’s systems. So from the earliest days we worked in an office in Cary while telecommuting and reporting to Encore’s parent engineering management organization in Wellesley Hills, Mass. Soon after Foundation became part of Encore Foundation got a Vax 11/780 running BSD Unix. That was such a sweet single user system. Unfortunately about a dozen and a half of us shared it and so we each got about an eighteenth of a Vax MIP: a truly miserable situation. It was slower than my first PC (a Southwest Tech 6800). I recall working funny hours for the sake of getting a bigger piece of the computing pie. However, we also got onto Arpanet with domain encore.com, and by virtue of that and our development charter Steve and I gradually accumulated access to hardware at other sites via Arpanet as we also accumulated local hardware resources. As I type this I have no foggy memory of the earliest hardware resources apart from the 11/780 and our dumb terminals. In those earliest days Encore outdid Amdahl for original funding and way way outdid Amdahl in profligate spending. I was disgusted to hear that the executive branch had hired dozens of salesmen way, way, WAY before having product to sell. What a cushy job I imagined, to visit prospective clients and spin tales of Encore’s products to be more than a year before the hardware transitioned from vaporware.

Suddenly in 1985 the insane burn rate caught up with Encore and they sold Foundation to Unisys. Steve’s reaction was to load up his hang gliders and drive to Oregon and back with wife and friend, flying their gliders at various places along the way. Meanwhile, I found the cheapest office space in Cary and filled it with the furniture and equipment we’d accumulated. By pure luck a byproduct of my previous startup, Network Products, had been creation of an inexpensive statistical multiplexer. This presented a set of eight asynchronous serial ports on each end of a synchronous leased phone line. I got a line and had one mux connected on the Massachusetts end (by this time at the Marlboro, MA site) with the other and connected at the Cary office. This gave us 960 bytes per second bandwidth in each direction. I got two of the Resolution terminals that put three logical screens on one very large physical screen with each logical screen connected to a separate mux port. The two remaining mux ports were connected to a system in Marlboro to drive a printer and a dialout modem on our end. On the Marlboro end we were connected to a 16 processor Multimax (Encore’s SMP product name in the early days) that was the beta test system named ?. Another Multimax hosted the Unix kernel and hardware developers and it was named Pinochio. Pinochio was the alpha test system and crashed many times a day while our beta system made it through the average day with only one or two crashes in the months after the Multimax hardware and the BSD port were first brought up. This was the ultimate “eat your own dog food” experience and the pressure on us getting the C compiler’s code generation right was enormous. One of the first parallel applications developed at Encore was our version of fsck. With 16 fsck processes running in parallel the filesystems could be fixed up fast to hasten the reboots after a crash!

Encore was intent on selling heavily into education markets and at the time that meant support for the Pascal language was a must have. BSD Pascal, created by Bill Joy and others, had major drawbacks from the point of view of Steve and I. In retrospect, while BSD Pascal genuinely stunk like a skunk, an imaginary better management should have told us to hold our nose and bundle it while turning attention to an implementation of Fortran that generated parallel code. But this assumed the imaginary management had a generous helping of precognition.

We got approval for development of a Pascal implementation using Oregon Software Pascal as a starting point while embracing Green Hills for its C compiler (and its Pascal that compiled the C compiler as it was itself a Pascal program but not standardized and so the Oregon Software compiler would not compile the Green Hills compilers). We got a source license from Oregon Software for free in return for giving them a new code generator and other improvements. Oregon Pascal had a very much stronger front end that would give students a fighting chance to correlate a compiler error message to the defect in their code that caused it and that would compile a great deal faster to give them merciful turn around times.

I got to write the back end of the compiler that translated the internal program representation into relocatable machine code targeting the National Semiconductor NS32k series of microprocessors. The NS32k architecture was the sweetest I’d ever encountered by a wide margin, being more regular than the Motorola 68k. One loopy detail that kept me busy was the fact that floating point literals in the instruction stream were stored in big endian byte order while the rest of the processor was little endian. Coupled with the fact that our development process required a lot of rehosting/retargeting steps where some hosts were big endian and some small I ended up with quite a collection of routines for manipulating instructions until we finally were running on the Encore machine and generating code directly for it. At this point the compiler was compiled by itself without any cross development tools. This compiler did not rely on an assembler: it output object files. To us this was a no brainer that gave a substantial performance edge. It’s funny how a large fraction of current main stream compilers go through an assembly phase. In fact I can’t think of a single one that compiles to machine code without using an assembler.

We finished the compiler and got it certified as ANSI compatible and it was an excellent tool for instruction at Encore sites like the University of Minnesota in Deluth. We had many friendly phone calls with the folks in Deluth and got first hand accounts of what it was like in the winter alongside Lake Superior (making my winters spent in Massachusetts seem like a equatorial vacations by comparison). Unfortunately for us all, Encore’s sales force was less than feeble and did not establish an education market that moved the sales needle.

But there was one interesting accomplishment stemming from the fact that Steve and I and others in the compiler group were able to keep entirely focus on our products: we ran out of bugs. Were were able to put out release notes that declared “no known bugs”. After more recently working on Java for years with a bug pile the size of a mountain it’s almost dream like to have been involved with nontrivial code bases with no bugs. Part of the reason why we got bug free and kept bug free was that we had regression test suite collections that were insanely large, and we required development of new tests for new features. And we had a big rule: nothing leaves the company without going through the regression tests. One of the Fortran tests was compiling and running a 737 simulator. I don’t recall how we got a copy of that, but I’m sure it wasn’t via a “front door”. We compiled the Pascal compiler through itself to multiple generations and compared the code generated for generation X with that of X-1. We were quality fiends in retrospect. It simply never got into our part of the Encore culture to ship broken software if we could avoid it and we did our best to avoid it.

For C, C++ and Fortran we ported Greenhills compilers that already had a 32k code generator. The C++ compiler was quite involved and luckily we had excellent folks in the group by that time like Jonathan Polito. We also ported Greenhills Pascal in order to compile the others. But Green Hills Pascal wasn’t even as attractive as BSD Pascal and so we never considered developing it into a product.

I just recalled one side trip I took in the very early days with Greenhills Fortran. For reasons that are long lost Greenhills supplied its own libm (math library) to to link against. Actually, the reason was clear: it was to win benchmarks. One reason it won benchmarks was that the transcendental functions (e.g. cosine) were written to execute fast. A math thing called a Taylor Series was used for these transcendentals and very few terms were used. The precision was terrible. It was so bad some of our earliest customers flagged it as a show stopper. So I found a decent book and rewrote parts of libm to have very much better precision while still runing as fast as possible. Benchmark performance suffered but our customer applications ran correctly.

One exciting piece of work we did with the Green Hills compilers was to make them automatically generate parallel code. That is, the compiler would see a loop, figure out the induction variable (the variable governing the iterations) and arrange to spawn threads of control for groups of iterations in parallel. There was some heavy duty global code analysis that had to happen to pull this off and I won’t go into it here. Steve did the heavy lifting while I was split between some of that development and more of the maintenance.

At some point it occurred to us to make the compilers themselves parallel. We split the compilers into five pieces and had each piece run on its own processor. This might possibly have led to publication. At the time Steve and I never gave any thought to publishing about our development work, but that’s another story. Anyway, our parallel compilers did not perform as expected. This turned out to be because our use of shared memory between the processes tickled a weakness in Encore’s BSD Unix kernel, causing thrashing (most likely of TLBs). At the time we had a vague understanding and called it the mystery overhead as we tried to get help from the kernel guys to understand it. We never figured a way around this. It’s real shame we didn’t publish our work as it was possibly novel.

One other thing that I did that I should have published about was to make the Pascal product automatically run user compilations in parallel with scheduling. Using some shared memory magic the compiler processes communicated with each other and agreed to when to execute compilations to get as many done as possible without overloading the system. So if there were twenty compiler users on an eight processor system I kept the “load factor” (number of concurrent active compiler processes) at or below some number that I’d empirically determined to be the “sweet spot” for throughput. I never even revealed that feature in the user documentation: it was just automatic and silent in its operation. This was 1986 or 1987.

Our part of the company responsible for the four languages grew from two people to a peak of 11 before shrinking down and going into caretaker mode.

Later we migrated from 32k to Motorola 88k products. I think that was the end of the road for the Oregon Software port, as I don’t remember them having an 88k back end and we didn’t create one. But at that point we were going after the simulation market with military contractors like Hughes Aircraft and nobody cared about Pascal in the field anymore.

At some point we upgraded the leased line to Marlboro from 9600 to 56k baud and with that boost of speed we were able to run X terminals. But we had a great deal of hardware in Cary by that time and were no longer at the mercy of the serial connection. The Network Products multiplexers never missed a beat. They just ran and ran, 24/7 until we were acquired by Sun Microsystems 12 years later. I was pretty satisfied as I’d written a large fraction of the firmware for those two boxes.

References:

Forest Earl Gilmore was the director of software development at Data General in the mid 1970s and my second level manager when I worked there working on a commercial language system. He was cofounder of Business Application Systems that spun out of Data General in late 1977 with a charter to make a portable operating system (“BASport”) and rich set of business applications that could be “written once and run anywhere” with the relatively easy ports of a virtual machine and kernel OS to each new hardware environment. Following that Earl cofounded Foundation Computer Systems, a developer of a fourth generation computer language designed to make application development a drag and drop experience. Encore Computer acquired Encore and later sold Foundation to Unisys, at which point Earl left to be an independent entrepreneur. Tragically a very aggressive cancer took Earl’s life when he was in his mid 40s. Earl was an extraordinary human being.

Cancer Trek Continues

In my 20s I occasionally went on solo backpacking trips. My most enthusiastic trip was from Coos Bay, Oregon to Eureka, California. I walked and camped down the Oregon coast and discovered that if you walk south in bright sun the first part of each day you turn brown on one side only. I don’t recall any fear on that trip. But on other trips into the White Mountains of New Hampshire and the Cherokee National Forest area of North Carolina I put myself into places where an unlucky accident could have serious consequences. One effect of being aware of this was to be damn careful where I put my feet, most especially avoiding places where snakes tended to hang out, keeping good balance when one step off trail was straight down 100 feet, etc. Another effect was to be exquisitely careful with navigation, having good maps, decent compasses, maintaining safe margins of water and taking good care of my feet. Oddly enough, apart from arranging to be noisy as I walked I was mostly oblivious in those days to the possibility of meeting bears and I don’t recall stringing my food and other smellables up and away from the tent, etc. Luck was on my side for that hazard!

But now I find myself on another trek involving the chance of a solitary consequence. I just finished a course of radiation treatment on the heals of chemotherapy treatment number two (hormone/chemo #1 ongoing). Currently the only proxy for my prostate cancer activity is the measure of an antigen in the blood called PSA. But my stage four cancer was diagnosed with a PSA of only 7.6, drastically lower than the specialists expected, meaning that the majority of my cancer activity does not involve cells having PSA on their surface. This is very bad, as although my PSA is now so close to zero that I’m declared “in remission” and “cancer free”, the truth is that my loved ones and I can only hope that the therapies annihilated the undetectable cancer cells along with the detectable ones. In theory this did happen, as the docetaxal chemo is highly toxic to the cancer througout the body and the radiation was directed at the sites were the (PSMA diagnostic) “visible” cancer resided.

The implication is that it may be difficult to impossible to detect recurrence of my cancer (and I have been told quite clearly that it will come back because it had metastasized already before diagnosis). If I’m lucky the cells correlated with a return of higher PSA levels will come back in concert with the undetectable ones, they will be found in the same places, etc, giving me a decent chance of getting treatment early before tumors begin to have serious side effects. However I don’t know yet if the situation is as bad as this: I’m hoping my next oncologist appointment will reveal an alternative proxy. But it’s been 17 months and surely I’d have been told by now. Except I hadn’t been told the basic situation with respect to undetectable cells by an oncologist: I had to get enlightenment from the science literature. It turns out that only about 1% of prostate cancers involve this “sneaky factor” and because of that there has been very little research done on dealing with this type. It’s on me to stimulate my current oncologist to become proactive about this or else help me find one who is willing, as “waiting and watching PSA” seems to be asking for too much of lady luck.

And apart from finding an alternate proxy or means of early detection of my sneaky cancer I also need to find out all I can about strategies for discouraging new growth. Now that the radiation is finished I expect my heart to bounce back (fewer/shorter PVC runs) and that will enable me to resume aerobics, etc. This is all while I eagerly watch the human trials of AOH1996, a potential silver bullet that might come on line in time for me.