No-code History: Frox a Scriptable SmartTV with a Magic Wand (1991)
Preface
It's hard to find content online about Frox, below are quotes from articles, books and a video presentation by Andy Hertzfeld, if you are interested in the programmable part jump straight to How Does it Work.
Below are all slightly edited quotes from the material listed in the Resources section, emphasis mine. My notes prefixed with đź’
Introduction
đź’ First person below is Hartmut Esslinger
In 1987, we started a new company called frox (as in “frog electronics”), with the goal of designing, developing, and producing a fully digital multimedia entertainment system. It was a truly visionary concept that just didn’t pan out.
Essentially, we wanted to integrate video-audio entertainment and computing into one system that would apply fully digital processing to all signals and data streams. Compared to our now decades-old concept, today’s “media centers” are still well behind the curve.
For two years, the venture consumed most of our attention and energy, until we realized that neither the company nor the market was ready for the concept.
What failed us in this undertaking wasn’t the raw-force/pure-play technology we were developing. Instead, we were undone by human failure—in both the overly “corporate” management team who over-politicized the venture and overspent its funding, and the investors who didn’t fully understand the painful process of applying high-tech capabilities to a consumer-focused product.
Interestingly, after Patricia and I left the venture, the investors continued frox with a new management team and new money. They succeeded at launching the prototype, but it ultimately failed because it was too expensive and unreliable.
đź’ First person below is Andy Hertzfeld
hi I'm Andy Hertzfeld and I've been working for the last year or so on developing an advanced user interface for Frox. A company involved with making the home entertainment system of the future.
I'm really excited about the system because I think it has the potential to create a revolution in the consumer electronics marketplace by combining a powerful computer as powerful as today's advanced workstations at the center of a complete home electronic system.
How Does it Work
Notice that Andy is using a mouse here, a computer without a keyboard ;)
In designing the Frox user interface the greatest challenge was to create a user interface that is appealing both to a technophobe and a technophile.
It's very hard to design an interface that is both simple and complex, so we solve the problem by providing complete end-user configurability.
What I'm going to show you now is how the end user can use the toolbox to build their own unique environment.
I can click on the command panel and over to the right I see I have this large toolbox.
Whenever you bring up the toolbox it means the system is kind of under construction.
I can move controls around just changing their positions, or I can customize them in various ways.
The toolbox itself consists of a bunch of boxes of parts.
There's actually over a thousand independent parts in the Frox system that the user can manipulate, by clicking on a box it opens.
If I click on different entities in the boxes such as this cuckoo noise you hear a cuckoo sound.
I'll take this cuckoo noise and drop it into the left half of a button makes the noise to reinforce it's being taken.
I'll grab the boing noise and drop it into the right of the switch, from now on this the switch will sound like Cuckoo ... Boing.
In a similar fashion we're in control of all the colors you see on the screen.
If I take this dab of blue and drop it here that panel becomes blue.
If I take this pink and drop it between the cracks it becomes pink.
I can change the color of the frog here to brown.
There's lots of other interesting parts in the toolbox browser.
You'll see that the looks of the controls are really independent of their functionality.
For example if I want to change the way this commercial switch looks but still make it a commercial switch I can choose the way I want it to look from any of these alternatives. Open one up take its shape image drop it on top of it, it'll change that one to be a different shape.
We've seen that we can change the looks of these controls but none of that really matters unless we can change their meaning.
The meaning of a control is encapsulated in these little nuggets of functionality called scripts.
In fact the system will come with hundreds of such scripts that can be dropped into any of dozens of different controls.
Let's look at the script associated with the stop CD button.
I can just click on that up, it opens up the script and we'll see that the script for the stop CD button is very simple.
Just is telling the CD to stop.
In a similar fashion this button here is an eject button when I press on it, it would eject the current CD.
I can get a different script that say is the play CD function and drop that into here now it becomes a play button, or now it would become a pause button.
By dropping in scripts I can change the meanings of any given control.
The real power comes in when end-users can design their own scripts.
I think it would be a good idea now to maybe write our own script from scratch so we can see how easy it is to to customize the system.
Let's turn this button here into a button so that instead of stopping the CD every time we press it it'll change the color of whatever panel we're in.
We can get up the toolbox browser, open up the script box and we'll see the special script with a lightning bolt.
When we drag this one out it will create an entirely new script.
We'll make a little program that will change the color of whatever panel we're in.
We'll use this pick operation which just picks one out of a box and then we'll put this box of colors next to it so we've effectively made it say pick a color.
We'll use the set color primitive to take that color we've picked and what we'll set with it is the color of the current panel.
There it is. We've just created a little program to set the color of the current panel.
We can take that script and drop it into a button here.
Put away the browsers and we'll see that when we press on the button it'll change the color of the panel we're in.
If we hold the button down it'll repeatedly execute the script changing their color repeatedly.
I'm really proud of the system because I think it has the potential to redefine how a user interacts with their audio and video environment.
It's a revolutionary system because it gives the end-user the same level of control over his environment that a programmer typically has.
The end-user is in control of every color sound and shape that they see on the screen.
The other most revolutionary aspect of the Frox system is that it's a completely open software based system.
Unlike traditional consumer electronic systems features can be added just by sticking in a floppy disk so the system never becomes obsolete.
The Frox system that an end-user buys in 1991 will be that much better in 1994.
I hope you've enjoyed watching this demo as much as I've enjoyed creating the system, thanks.
Frox Overview
I'm going to launch it here and while it's launching make a couple apologies the main one being that the computer here has only 8 bits per pixel so it can only display 256 simultaneous colors whereas our real system will have 24 bits per pixel and be able to display 16 million colors.
The main way the user interacts with the system is through what we call a magic wand pointing device much like a normal remote control but only one button on it and the user will point at the screen and this hand on the screen tracks the movement of the magic wand.
Your way of manipulating the environment is by clicking on these little controls with your hand.
You'll see that I can grab the volume control and will actually make it louder.
I can move the balance control to listen to just the left or just the right.
I have a wide variety of other controls that I'll be showing you later.
Probably the most important panel here is the switch box panel which shows you through graphics and animation all the activity currently going on in the system.
If we activate the CD we'll see the notes emanating from the CD when I hit pause on the CD will notice that the CD stops spinning.
When I start it up again as it spins in real life it spins in the switch box panel.
In a similar fashion if I pause the VHS cassette it will pause, when I play it, will begin to animate again.
We can switch between different screens using the command panel at the bottom.
One click brings it up, there's a push button here to dismiss it and then just clicking on an image of a screen will take us to that screen.
We'll look at a large video screen it's paused here, we can get it going or we can go to a variety of other screens.
One of the most unique and extraordinary benefits of the Frox system is the way it deals with your media such as your CDs.
You can select the individual CDs by their album cover, I can go to a screen where the album covers are displayed pretty large.
As we click we can see a variety of different album images corresponding to each CD that's currently accessible to the user.
I can even go to another screen here that has a very large panel so you can see every song on the CD to play, displayed all at once.
Now we're playing "Like a Rolling Stone" if we want to play "Ballad of a Thin Man" we just click on it or "Just Like Tom Thumb's Blues" or "Desolation Road".
That's the CD capability. I'll turn off the CD player now so it'll be a little easier to show you other dimensions of the system.
Reports at the Time
đź’ From Chicago Tribune article
The FroxSystem revolves around a custom-designed Sun Microsystems computer workstation. A workstation is a personal computer on steroids. This computer controls and manipulates all of the audio and video in the system. It converts signals from analog to digital and processes them in the digital domain.
The FroxSystem learns all of the infrared remote control commands of your existing audio and video sources. It then takes control of the entire system with a unique one-button wireless remote called the FroxWand that operates like a flying computer mouse. Pressing a button brings up a display of the TV screen of controls for the piece of equipment you wish to operate.
The FroxVision monitor manipulates more than 360,000 pixels 60 times a second, 40 times the industry standard.
Frox offers a choice of a 31-inch direct view monitor, a 52-inch rear projection monitor, or a monster 10-foot front projection monitor.
Since computer software operates the advanced digital hardware of the FroxSystem, the system can be updated and improved without replacing the hardware. Frox supplies updates on VHS videocassettes. However, there`s an even easier method. Frox made an arrangement with satellite program provider Turner Broadcasting Co. to transmit updates invisibly on superstation WTBS, which is carried on satellite and nearly all cable systems. A portion of the TV picture you can`t see, called the vertical blanking interval (VBI) contains room for additional data.
đź’ From CNN article
The TV is the focal point of the system, but what makes it all work is a built-in computer as powerful as an engineering workstation. Soon the machine will simultaneously monitor electronic databases for news or other information of particular interest, answer the telephone, watch for incoming electronic mail, and control additional home appliances even as it runs the TV or stereo. In essence, the Frox machine is an ambitious effort to give the boob tube some real smarts.
The goal: desktop video computers that users interact with, not merely another box for couch potatoes to sit and stare at. These video computers would usher in video encyclopedias and other interactive educational and training tools. They could read and display patterns of stock price quotes, and would make possible hundreds of new and elaborate computer games. Ultimately just about anybody will be able to create electronic productions that mix snippets of moving video and sound with conventional text and computer graphics. FOR EXAMPLE, you could write your mother a letter including video highlights of your daughter's birthday party or your trip to Europe, with commentary dubbed in. You would mail it to her on a single computer disk -- or, better yet, transmit it to her computer almost instantly over telephone lines. One day, video computers may even act as the futuristic videophones that telecommunications companies have promised for decades but never really delivered.
Moreover, many experts aren't so sure that ordinary people can master the exacting techniques necessary to put together a comprehensible video program, even if it's just an edited home movie. Indeed, some contend that most people would really only need or want a 'multimedia player' -- a TV or computer that allows them more control over prerecorded, professionally produced interactive video programming.
Frox was founded last year by Hartmut Esslinger, a West German whose Frogdesign firm helped devise the striking ergonomic look of most of Apple's personal computers and the Next machine.
Steve Jobs helped Esslinger refine the idea last year but had to back out to devote full time to Next. Esslinger continued on his own. To build the prototype he enlisted the help of Andreas Bechtolsheim, one of Sun Microsystems' founders; Peter Costello, another top Sun engineer; and Hertzfeld.
Trivia
If a comment in the youtube video is to be believed: "Andy developed the Frox prototype in (object-oriented!) assembly language on a processor family he KNEW wouldn't be used in the final product -- thus ensuring that the actual product wouldn't be a hacked-up expansion on the bones of the prototype."