« October 2007 | Main | April 2008 »

November 29, 2007

My @media Ajax Keynote

Herewith a hacked-up version of my S5 slides (UPDATED URL), with notes and commentary interpolated at the bottom of each slide.

See how JS is paired with Flash -- poor, mundane HTML, CSS, DOM! HTML5 needs a new name.

I described Doug Crockford as "the Yoda of Lambda JavaScript programming" at a mid-2006 talk he invited me to present at Yahoo!, so I thought I would start by riffing on whether the role still fits. So far, so good.

But three prequels later, the outlook for Yoda is not good. Large, heavy, spinning, flying-saucer-proprietary runtimes are hurtling toward him!

Enough Star Wars -- Doug's too tall for that part (even if it's only a muppet). Let's try a taller wizard...

I don't really believe ES4 is a demon from the ancient world, of course. I'm afraid the JS-Hobbits are in trouble, though. As things stand today, Silverlight with C# or something akin (pushed via Windows Update) will raze the Shire-web, in spite of Gandalf-crock's teachings.

I can roleplay too: let's see, renegade veteran from the losing side of an epic war against an evil empire... yeah, I can relate.

I really do think that JS's multi-paradigm nature means there is no one-true-subset for all to use (whether they like it or not), and the rest -- including evolutionary changes -- should be kept out. I reject the idea that instead of making JS better, programmers should somehow be made "better". The stagnation of JS1, and "little language" idolatries surrounding it (those one-true-way JS subsets), impose a big tax on developers, and drive too many of them away from the web standards and toward WPF, Flex, and the like.

Ok, enough role-playing geek fun -- let's get down to brass tacks: what's really going on with the ES4 fracas?

Clearly ES4 will be tough to standardize. Standards often are made by insiders, established players, vendors with something to sell and so something to lose. Web standards bodies organized as pay-to-play consortia thus leave out developers and users, although vendors of course claim to represent everyone fully and fairly.

I've worked within such bodies and continue to try to make progress in them, but I've come to the conclusion that open standards need radically open standardization processes. They don't need too many cooks, of course; they need some great chefs who work well together as a small group. Beyond this, open standards need transparency. Transparency helps developers and other categories of "users" see what is going on, give corrective feedback early and often, and if necessary try errant vendors in the court of public opinion.

Given all the challenges, the first order for ES4 work is to finish the reference implementation and spec writing process, taking into account the ongoing feedback. Beyond that, and I said this at the conference, I believe we need several productized implementations well under way, if not all but done, by the time the standard is submitted for approval (late 2008). This will take some hard work in the next ten months.

My hope is to empower developers and users, even if doing so requires sacrifice on the part of the vendors involved.

Some assert that JS1 is fine, browsers just need better APIs. Or (for security), that JS1 with incompatible runtime semantic changes and a few outright feature deletions is fine, but mainly: browsers just need better APIs. Or that Security comes first, and the world should stop until it has been achieved (i.e., utopia is an option). But I contend that JS must improve along with browser APIs and security mechanism and policy, both to serve existing and new uses, and to have a prayer of more robust APIs or significantly better security.

It's clear from the experiences of Mozilla and security researchers I know that even a posteriori mashups built on a capability system will leak information. So information flow type systems could be explored, but again the research on hybrid techniques that do not require a priori maximum-authority judgments, which do not work on the web (think mashups in the browser without the user having to click "OK" to get rid of a dialog), is not there yet. Mashups are unplanned, emergent. Users click "OK" when they shouldn't. These are hard, multi-disciplinary research problems.

Where programming languages can help, type systems and mutability controls are necessary, so JS1 or a pure (semantics as well as syntax) subset is not enough.

I am personally committed to working with the Google Caja team, and whoever else will help, to ensure that JS2 (with the right options, and as few as possible) is a good target for Caja. The irony is that when combined with backward compatibility imperatives, this means adding features, not removing them (for example, catchalls).

A note on names: I used JS2 in the title and the slides to conjure with the "JS" and "JavaScript" names, not to show any disrespect to ES/ECMAScript. All the books, the 'J' in AJAX (when it's an acronym), the name of the language dropped most often at the @media Ajax conference, all call the language by the "JavaScript" name. Yeah, it was a marketing scam by Netscape and Sun, and it has a mixed history as a brand, but I think we are stuck with it.

This slide compresses a lot, but makes some points often missed by fans of other languages. Browsers will always need JS. Browsers cannot all embed the C Python implementation, the C Ruby implementation, etc. etc. -- code footprint and cyclic leaks among heaps, or further code bloat trying to super-GC those cycles, plus all the security work entailed by the standard libraries, are deal killers.

The multi-language, one-runtime approach is better, but not perfect: IronPython is not Python, and invariably there is a first-among-equals language (Java on JVMs, C# on the CLR). We are investing in IronMonkey to support IronPython and IronRuby, and in the long run, if everyone makes the right moves, I'm hopeful that this work will pay off in widespread Python and Ruby support alongside JS2. But it will take a long while to come true in a cross-browser way.

Silverlight is not able to provide browser scripting languages in all browsers. Even if IE8 embeds the DLR and CLR, other browsers will not. Note the asymmetry with ScreamingMonkey: it is likely to be needed only by IE, and only IE has a well-known API for adding scripting engines.

The "too small" and "too few primitives" points remind me of Guy Steele's famous Growing a Language talk from OOPSLA 1998 (paper). If you haven't seen this, take the time.

During the panel later the same day, Jeremy Keith confronted me with the conclusion that JS2 was pitched only or mainly at Java heads. I think this slide and the next gave that impression, and a more subtle point was lost.

I hold no brief for Java. JS does not need to look like Java. Classes in JS2 are an integrity device, already latent in the built-in objects of JS1, the DOM, and other browser objects. But I do not believe that most Java U. programmers will ever grok functional JS, and I cite GWT uptake as supporting evidence. This does not mean JS2 panders to Java. It does mean JS2 uses conventional syntax for those magic, built-in "classes" mentioned in the ES1-3 and DOM specs.

In other words, and whatever you call them, something like classes are necessary for integrity properties vital to security in JS2, required for bootstrapping the standard built-in objects, and appropriate to a large cohort of programmers. These independent facts combine to support classes as proposed in JS2.

JS1 is used by non-programmers, beginning programmers, "front end designers". It is copied and pasted, or otherwise concatenatively programmed, with abandon (proof: ES4 has specified toleration of Unicode BOMs in the middle of .js files! How did those get there?). This was a goal, at least in the "pmarca and brendan"@netscape.com vision circa 1995, over against Java applets. We succeeded beyond our wildest nightmares.

Netscape 2 had some very clueful early adopters of JS (Bill Dortch, if you are reading this, leave a comment). Years later, Doug Crockford led more select hackers toward the right end of the distribution, but much of the middle was necessarily bypassed: you can't reach this cohort without taking over Java U.

What's more, I observe that the Lambda-JS Jedi order is inherently elitist (I'm not anti-elitist, mind you; natural elites happen in all meritocratic systems). For many current Knights, it must remain so to retain its appeal.

Now, there's nothing wrong with using closures for (partial) integrity and prototypes for inheritance; I like these tools (I should, I picked them in a hurry in the early days). But really, why should everyone be required to learn the verbose, error-prone, and inherently costly functional-JS-OOP incantations (power constructors, module patterns, etc.), instead of using a few concise, easier to get right, and more efficient new declarative forms that JS2 proposes?

It's not as if JS2 is renouncing prototypes or closures in favor of "the Java way". That's a misinformed or careless misreading. Rather, we aim to level the playing field up, not down. JS2 users should be able to make hardened abstractions without having to write C++ or create Active X objects. And power-constructor and module pattern fans can continue to use their favorite idioms.

JS1 favors closures (behavior with attached state) over objects (state with attached behavior) with both more abstraction (because names can be hidden in closures) and greater integrity (because var bindings are DontDelete). While JS1 is multi-paradigm, going with the grain of the design (closures over objects) wins. In my talk, I acknowledged the good done by Doug and others in teaching people about functional programming in JS.

However, there are limits. JS1 closure efficiency, and outright entrainment hazards that can result in leaks, leave something to be desired (the entrainment hazards led Microsoft to advise JScript hackers to avoid closures!). You could argue that implementations should optimize harder. Arguing is not needed, though -- high quality optimizing runtime, which fit on phones, are what's needed.

Beyond efficiency, using closures for modules and class-like abstractions is verbose and clumsy compared to using new syntax. Dedicating new syntax is scary to some, but required for usability (over against __UGLY__ names), and allowed under script type versioning. In the absence of macros (syntactic abstraction), and so long as macros can be added later and syntactic sugar reimplemented via macros at that time, my view is that we should be generous with syntactic conveniences in JS2.

So JS2 has more syntax (user interface) as well as substance (complementary primitives). This could be a burden on people learning the new language, but I think not a big one. In practice over the next few years, the bulk of the focus in books and classes will be on JS1 and Ajax. A programming language for the web should be a many-storied mountain, and most users will not ascend to the summit.

The main burden of new syntax is on folks writing source-to-source translators, static analyzers, and the like. These more expert few can take the heat so that the many can enjoy better "user interface".

To respond to the contrarian arguments in the previous slide:

• Whoever prefers a subset is free to continue using it on JS2 implementations. If your pointy-haired boss imposes class-obsessed B&D programming on you, get a new job.
• DOM profiles show high dynamic dispatch, argument/result conversion, and other costs imposed by untyped JS1 in current implementations. Better implementations and JS2 usage can help.
• Security is best defined as end-to-end properties that must be engineered continuously according to economic trade-offs as the system evolves. Utopia is not an option.
• Some people are learning, but many others are not, and vendors sell C# and AS3 against JS1 for good reason.

When I hear concerns that ES4 will "break the web," I can't help but think of how many times I've heard that the web is already broken! The risks of not adopting ES4 surely must factor into this calculus, too.

At the very least, don't put all eggs in the "make people better" basket.

I get all kinds of fan-mail :-/.

The argument that stagnation on the web fostered Ajax and Web 2.0 is false. Only when XMLHttpRequest was cloned into other browsers, and especially as Firefox launched and took market share back from IE, did we see the sea-change toward web apps that rely on intensive browser JS and asynchronous communication with the server.

Like Mal Reynolds, I really do believe these things -- not that I'm ready to die for my beliefs ("'Course, that ain't exactly plan A").

There won't be a single browser release after which all power shifts to web developers writing in JS2. The process is more gradual, and it's easy to forget how far we've come already. We're well along the way toward disrupting the desktop OSes. Yet JS2 on optimizing VMs will liberate developers in ways that JS1 and plugins cannot.

I lived through the Java dream. Netscape was building an optimizing JIT-compiling runtime in 1996 and 1997, while Sun acquired Anamorphic and built HotSpot. The layoffs at the end of 1997 brought all the Netscape Java work crashing to a halt, and caused the "Javagator" rendering engine team to reinvent their code in C++ as Gecko (originally, "Raptor").

In spite of all the hype and folly, the dream could have come true given both enough time and better alignment (including open source development) between Sun and Netscape. A lot more time -- Java was slow back then, besides being poorly integrated into browsers.

There are many ironies in all of this. Two I enjoy often:

• Flash filled the vacuum left by the decline of Java in the browser, and now provides a vector for JS2 on Tamarin.
• Microsoft dumped Java, depriving Outlook Web Access of an asynchronous I/O class, wherefore XMLHttpRequest.

If you take only one point away from this talk (I said), it should be that type annotations are not required for much-improved pure JS performance.

Tracing JIT Benchmarks

This chart shows results, normalized using SpiderMonkey performance at unity (so taller is faster), for the JavaGrande benchmark ported to JS (untyped JS except where noted: "Tamarin with Type Annotations"). The "Trace-Tree JIT" blue bars show results for a clever translation of Tamarin bytecode into Java bytecode (with runtime type helpers) fed into a tracing JIT implemented in Java(!). Amazingly, this approach competes with Rhino and Tamarin, even Tamarin run on typed-JS versions of the benchmarks.

The Crypt benchmark could not be run using the trace-based JIT at the time this chart was generated.

Tracing JIT Benchmarks (2)

More good results, especially given the preliminary nature of the research. With an x86 back-end instead of the Java back-end used for these benchmarks, and further tuning work, performance should go up significantly. Even at this early stage, Series, SOR, and SparseMatMult all show the tracing JIT working with untyped JS beating Tamarin on typed-JS versions of these benchmarks.

In response to an informal recap of my presentation the other day, Rob Sayre mentioned Peter Norvig's presentation on design patterns in dynamic programming. This caused me to flash back to the bad old days of 1998, when certain "Raptor" architects would wave the Gamma book and make dubious assertions about one-true-way design patterns in C++.

Norvig's slides show what was lost by the swerve toward static, obsessively classical OOP in C++ and Java, away from dynamic languages with first-class functions, first-class types, generic methods, and other facilities that make "patterns" invisible or unnecessary. JS2 aims to restore to practical programmers much of what was lost then.

The Romans called wheat integrale, referring to the potent and incorruptible completness of the kernel. Integrity as a security property is not far removed from this sense of completeness and soundness. JS1 simply lacks crucial tools for integrity, and JS2 proposes to add them.

The following slides (I've coalesced multiple slides where possible) show the evolution of a webmail library from JS1 to JS2, via gradual typing, in order to increase integrity and simplify code, avoiding repetitious, error-prone hand-coded latent type checking. The transport code is omitted, but you can see JSON APIs being used for transfer encoding and decoding.

function send(msg) {
  validateMessage(msg);
  msg.id = sendToServer(JSON.encode(msg));
  database[msg.id] = msg;
}

function fetch() {
  handleMessage(-1);                  // -1 means "get new mail"
}

function get(n) {
  if (uint(n) !== n)                  // JS1: n>>>0 === n
      throw new TypeError;
  if (n in database)
      return database[n];
  return handleMessage(n);
}

var database = [];

function handleMessage(n) {
  let msg = JSON.decode(fetchFromServer(n));
  if (typeof msg != "object")
      throw new TypeError;
  if (msg.result == "no data")
      return null;
  validateMessage(msg);
  return database[msg.id] = msg;
}

function validateMessage(msg) {
  function isAddress(a)
      typeof a == "object" && a != null &&
      typeof a.at == "object" && msg != null &&
      typeof a.at[0] == "string" && typeof a.at[1] == "string" &&
      typeof a.name == "string";
  if (!(typeof msg == "object" && msg != null &&
      typeof msg.id == "number" && uint(msg.id) === msg.id &&
      typeof msg.to == "object" && msg != null &&
      msg.to instanceof Array && msg.to.every(isAddress) &&
      isAddress(msg.from) && typeof msg.subject == "string" &&
      typeof msg.body == "string"))
      throw new TypeError;
}

It's rare to see anything in real-world JS like the detailed checking done by validateMessage. It's just too tedious, and the language "fails soft" enough (usually), that programmers tend to skip such chores -- sometimes to their great regret.

type Addr = { at: [string, string], name: string };
type Msg = {
  to: [Addr], from: Addr, subject: string, body: string, id: uint
};

function send(msg: like Msg) {
  msg.id = sendToServer(JSON.encode(msg));
  database[msg.id] = msg;
}

function fetch()
  handleMessage(-1);

function get(n: uint) {
  if (n in database)
      return database[n];
  return handleMessage(n);
}

function handleMessage(n) {
  let msg = JSON.decode(fetchFromServer(n));
  if (msg is like { result: string } && msg.result == "no data")
      return null;
  if (msg is like Msg)
      return database[msg.id] = msg;
  throw new TypeError;
}

Important points:

• Structural types are like JSON, but with types instead of values
• The like type prefix makes a "shape test" spot-check
• Note how fetch is now an expression closure
• No more validateMessage! Structural types ftw! :-)

type MsgNoId = {
  to: [Addr], from: Addr, subject: string, body: string
};

function send(msg: like MsgNoId) {
  msg.id = sendToServer(JSON.encode(msg));
  database[msg.id] = copyMessage(msg);
}

function handleMessage(n) {
  let msg = JSON.decode(fetchFromServer(n));
  if (msg is like { result: string } && msg.result == "no data")
      return null;
  if (msg is like Msg)
      return database[id] = copyMessage(msg);
  throw new TypeError;
}

function copyMessage(msg) {
  function newAddr({ at: [user, host], name })
      new Addr([user, host]: [string, string], name);

  let { to, from, subject, body, id } = msg;
  return new Msg(to.map(newAddr), newAddr(from), subject, body, id);
}

This stage copes with a confused or malicious client of the webmail API, who could mutate a reference to a message to violate the validity constraints encoded in stage 1's validateMessage.

The MsgNoId type allows the library client to omit a dummy id, since send initializes that property for the client.

Note the righteous use of parameter and let destructuring in copyMessage.

function send(msg: like MsgNoId) {
  msg.id = sendToServer(JSON.encode(msg))
  database[msg.id] = msg wrap Msg
}

function handleMessage(n) {
  let msg = JSON.decode(fetchFromServer(n))
  if (msg is like { result: string } && msg.result == "no data")
      return null
  return database[msg.id] = msg wrap Msg
}

wrap is both an annotated type prefix and a binary operator in JS2. It makes a wrapper for an untyped object that enforces a structural type constraint on every read and write, in a deep sense. So instead of copying to provide integrity through isolation, this alternative third stage shares the underlying message object with the library client, but checks all accesses made from within the webmail library.

Notice the lack of classes so far.

UPDATE: A revised and extended version of this evolutionary arc is now available as a tutorial at ecmascript.org, with compelling side-by-side comparisons of successive stages.

The "typed APIs with untyped code" pattern is particularly winning in our experience building the self-hosted built-ins in the ES4 reference implementation.

During this slide, I shot screaming slingshot flying monkeys (complete with little black masks and capes) into the audience. I'm sorry I could bring only a handful on this trip!

Chris provides source code, Firefox 3 alpha builds, and Ogg video/audio files.

Now available as a Firefox 3 addon!

Posted by brendan at 2:36 AM | Comments (6)

November 5, 2007

ES4 News and Opinion

ES4 News

Some news on language size: the proposed ECMAScript 4th edition (ES4) grammar is a bit more than twice as big as ES3's, counting several ways (concrete productions, abstract syntax tree node types):

Syntax                   ES3   ES4
------                   ---   ---
Concrete                 238   490
  Type Exprs                    50
  Classes & Interfaces          23
  Patterns                      17

Abstract                  33    77

This is not out of line given the eight years since ES3, which came less than three years after ES1. Much of ES4 has been developed and shipped in ES3-based languages: AS3 in Flash, JS1.6 and 1.7 in Firefox, Opera's extensions including array destructuring. And we're being generous with syntactic conveniences, which desugar to a smaller core language.

On the compatibility front, we have a new document on known incompatibilities between ES3 (ECMA-262 Edition 3 the spec, not real-world behavior of ES3 implementations in browsers -- note well) and ES4.

Contradiction

Chris Wilson: "... I don't think the other Microsoft guys ever intended to say 'everything about ES4 is bad'."

Pratap Lakshman (Microsoft): "We do not support or agree to the current ES4 proposal, either in whole or in part."

Individual intentions can be hard to judge, and anyway matter less than effects. My point is not to guess about intentions, only to point to statements and actions.

Clarification

It's a relief to learn that Chris's charge of "shouting down dissent" -- confusingly juxtaposed with his words "I think it's a shame that dissenting opinion has been hidden from view", which seem to point to the ECMAScript standards body's majority in a way that left many readers thinking that someone in Ecma TC39-TG1 actually shouted at Microsoft reps during a meeting -- was really just a complaint about bloggers "shouting" (which is news-at-11 fodder, don't you think? I've seen shouting from all sides on this one).

That leaves the charge of "ignoring", which was not aimed at the blogosphere -- that accusation was aimed straight at "the TG-1 committee chair", presumably me (I'm TG1 Convenor). But it is simply false to claim I've ignored compatibility concerns or anything else raised by a member of TG1.

Still, if we can make progress by agreeing that bloggers sometimes shout, or show bad manners, I'm all for it. My manners aren't perfect, and I'm sorry when they fail. But morals trump manners, and the overarching issue before us remains: is it right for ES4 to be stalled in committee, or for ES3 to be forked into a now-hidden "ES3.1", because Microsoft and Doug Crockford object to ES4 on general grounds?

Consistency

Whatever you think of ES4 (and it's not done yet -- we just released a reference implementation update), the history nicely reconstructed by Neil Mix should make clear that it has been a long time coming, most of the time without anyone refusing to support it "in whole or in part". The July 2006 meeting notes include a statement from Pratap that ES4 would be implemented in the next major IE release after IE7.

Of course this all changed, starting early in 2007, officially only in March. But why did it change? Was all the pre-history, including Microsoft's implementation of JScript.NET, a big mistake? Did JScript.NET really "fail", and fail in any way that bears on ES4 itself or its future on the client side of the web? I think not, and the generalities about breaking the web, or the web not needing to change much, do not explain this about-face from 2006 to 2007.

Anyone who reads the face-to-face meeting notes can see that everyone working on ES4, including me, would work on topics of interest with whomever attended meetings. And we worked together well enough when focused on technical issues. Allen Wirfs-Brock of Microsoft, in particular, is very knowledgeable and insightful.

Rejection

But the cold hard fact, documented in that September face-to-face meeting note, is that the dissenting minority in TG1 rejects ES4 in whole and in part. There does not seem to be room for compromise.

As TG1 Convenor, I've tried to keep any "3.1" work in the context of ES4, to avoid a fork of ES3 into incompatible derivative languages. In particular, any ES3.1 should be a subset of the ES4 reference implementation, so it can be tested. This seemed agreeable in March, and Allen reiterated agreement in April.

Since April, it has been impossible for the majority of TG1 to evaluate ES3.1. There has been no work on the wiki, and almost no communication to the majority of TG1 about what's in ES3.1 or how it will be tested for backward compatibilty with ES3 and forward compatibility with ES4.

Do The Right Thing

Neither a hidden and unready alternative proposal, nor general objections from a minority in the group, should halt a standard in progress. ES4 should be delayed or abandoned only on technical demerits, demonstrated specifically. There are enough implementors working together to make a standard, and ES4 as it has evolved has always been the expected successor standard. To block it based on general fears and opinions prejudges both the final standard and the market for it. The market, not Ecma TC39-TG1, should decide ES4's fate.

It should be clear by now that the majority of TG1, who favor ES4, want to keep the group from splitting, or otherwise risking incompatible forks of ES3. Everyone loses in that scenario.

We wouldn't have spent all the time in the wiki and at meetings if we all, including me, didn't want a win-win outcome for all players. And I personally suggested to Chris in March that Microsoft could be a big winner by backing ES4 and supporting it in IE.

Is it still possible for everyone to win? I have to say I'm not as positive as I used to be. If we all work on testable compatibility via the reference implementation, then I'm somewhat more hopeful. But it will take a better stance than rejection of ES4 "in whole [and] in part" to get somewhere.

Posted by brendan at 10:47 PM | Comments (3)