Well, I'm Back

« Light-Weight Instrumentation from Relational Queries Over Program Traces | Main | Travel »

May 13, 2005

Rendering Web Page To Images

thumbviewSidebar.xul

<?xml version="1.0"?>

<?xml-stylesheet href="chrome://global/skin" type="text/css"?>

<window
    windowtype="global:thumbviewSidebar"
    onclick="update()"
    id="win"
    xmlns:html="http://www.w3.org/1999/xhtml"
    xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
  <script type="application/x-javascript" src="chrome://thumbview/content/thumbview.js"/>
  <vbox flex="1" id="before"/>
  <html:canvas id="canvas"/>
  <vbox flex="1" id="after"/>
</window>

thumbview.js

function update() {
  var w = content.innerWidth + content.scrollMaxX;
  var h = content.innerHeight + content.scrollMaxY;
  if (w > 10000) w = 10000;
  if (h > 10000) h = 10000;

  var container = document.getElementById("win");
  var canvasW = container.boxObject.width;
  var scale = canvasW/w;
  var canvasH = Math.round(h*scale);

  var canvas = document.getElementById("canvas");
  canvas.style.width = canvasW+"px";
  canvas.style.height = canvasH+"px";
  canvas.width = canvasW;
  canvas.height = canvasH;
  var ctx = canvas.getContext("2d");
  ctx.clearRect(0, 0, canvasW, canvasH);
  ctx.save();
  ctx.scale(canvasW/w, canvasH/h);
  ctx.drawWindow(content, 0, 0, w, h, "rgb(0,0,0)");
  ctx.restore();
}

var NavLoadObserver = {
  observe: function(aWindow)
  {
    update();
  }
};

function start() {
  var obs = Components.classes["@mozilla.org/observer-service;1"].
    getService(Components.interfaces["nsIObserverService"]);
  obs.addObserver(NavLoadObserver, "EndDocumentLoad", false);
}

window.addEventListener("load", start, false);

Update! I overhauled this entry significantly since we may not be adding a method to 'window' after all. The drawWindow method will be there though.

Posted by roc at May 13, 2005 6:01 PM

Comments

A feature that has been missing for over 4+ years. Great work Roc!

Posted by: Doug Turner at May 13, 2005 8:19 PM

Great to see this implemented! I'm sure we'll see loads of interesting extension ideas around this...

On a remotely related issue: Any chance you can fix https://bugzilla.mozilla.org/show_bug.cgi?id=204278 (XUL elements cannot be stacked on top of browser element) for Gecko 1.9, too? IMO, that's another bug that, when fixed, might have great impact on extensions... For example, Mouse Gestures might be able to stack a transparent canvas element on top of the browser, and display actual drawn mouse trails instead of inserting hundreds of tiny HTML elements into the web page (which is the only way to do it right now).

Posted by: jens.b at May 13, 2005 8:44 PM

Awsome.

* Showing thumbnails of pages on the back & forward dropdown list
* Generating thumbnails of prefetched pages on google and showing them inline (greasmonkey style extension)
* A bugzilla helper extension, that would let you take a "screenshot" of the place in the page you feel is being mis-rendered.
* Bookmark gallery. When hitting ctrl+b the sidebar would not only show the bookmarks, but images of the pages too.

Another thing that I'd like to see is a binary application or an easy to use library that would generate images for you.

This could be a nice addition to gaim for example, which could show a thumbnail of a persons site in the "more information" dialog. Also any website could generate screenshots for their sites (see: http://dev.upian.com/hotlinks/)

Posted by: mvir at May 13, 2005 8:46 PM

Great, now I would like to see a thumbnail in the history, instead of the pagetitle only.
That would be a great feature ... it is easier to remember a visited page by seeing the thumbnail than only the title ... extension idea (I would make it myself if i could) ?

Posted by: Yoeri at May 13, 2005 9:21 PM

This is a _superb_ work, roc. In a former professional life, I was the CTO of a web agency and I can tell you web agencies are going to _love_ you if they can generate and save (can you save the bitmap right now?) such a thumbview from within the browser, w/o external tool.
Once the core is checked in, I strongly recommend making your extension an official extension to Firefox released by the Mozilla Foundation. This is awesome.

Posted by: Daniel Glazman at May 13, 2005 9:29 PM

The most immediate thought for an extension that comes to mind: replacing the tab bar with a tab-thumbnail side bar.

Posted by: Axord at May 13, 2005 9:58 PM

jens.b: that should get fixed in 1.9, but not 1.8.

mvir, glazman: we should be able to do what you want once we have the ability to save the contents of a canvas. The plan is to have a .toDataURL() method that returns a data: URL containing a base64 encoded PNG. We should be able to get that into 1.8.

One thing that's not obvious here is that windows can be transparent, and drawWindow respects/captures the transparency. (Use "rgba(0,0,0,0)" as the background color.)

Another use of drawWindow is that it extends the power of the canvas element by giving you a way to conveniently draw certain kinds of content into a canvas --- e.g., text, or even SVG!

Posted by: Robert O'Callahan at May 13, 2005 11:21 PM

But ? That means that future versions of Minimo could have ... full zoomable pages as Opera ?

Aooouuuuuuuuhhhhhh !

Posted by: Da Scritch at May 13, 2005 11:36 PM

no. interactive zooming requires different technology. That's where our cairo work comes in.

Posted by: Robert O'Callahan at May 13, 2005 11:45 PM

This is what I meant:
http://apple.slashdot.org/apple/04/03/31/0513245.shtml?tid=126&tid=185&tid=95
http://www.acm.uiuc.edu/macwarriors/projects/trailblazer/

Posted by: Yoeri at May 14, 2005 1:42 AM

Excellent! This paves the way to automatic site-map generators (to make posters to stick on one's wall).

Stick a spider and thumbnailer onto one of the graph layout packages and much fun can be had
I had a rough one going with KHTML and graphviz once, but gave up because at the time KHTML didn't render well enough. But Gecko doing it would rock!

- Colin

Posted by: Colin Coghill at May 14, 2005 1:55 AM

Wouldn't it be cool to have the whole page in miniature in the sidebar with a rectangular border marking the visible portion of the page (the part that's in the main window). Then you could scroll around the page by moving that rectangular border around the thumbnail.

Posted by: Rory Parle at May 14, 2005 2:22 AM

I think i want to use this for reporter in the future.

This looks like it has so much potential for tracking layout regressions.

Awesome work roc!

Posted by: Robert Accettura at May 14, 2005 6:42 AM

Any idea when this will be present in Nightlies?

Posted by: Jed at May 14, 2005 12:19 PM

What I want, on a related note, is some way I can use Gecko to replace html2ps!

Posted by: Michael Newton at May 14, 2005 6:39 PM

will you add a way to let extension authors know when a page is completely done loading, i.e. when the background images are there as well?

Posted by: christian biesinger at May 15, 2005 1:28 AM

Will this work on a Linux box without an X server? Or do you need an X server to do the pixel rendering? Sorry if this is a naive question, but it's a huge problem when trying to create thumbnails with khtml.

Posted by: Nelson at May 15, 2005 6:55 AM

jed: within a few days, perhaps

michael: someone could write a remote-controllable XUL app for this

biesi: me? no. But I'm sure you could do this yourself :-). (BTW the GECKO_FORCE_PAINT_ONLOAD environment variable forces onload to be blocked by CSS background image loads.)

Nelson: It needs X. At some point we'll land a version of tetron's null-widget code so that you can run apps without X.

Posted by: Robert O'Callahan at May 15, 2005 10:32 AM

Wow very nice indeed, if only we could have Omniweb like Tabs for FireFox, that would be really awesome.

Somebody needs to get onto this and fast!

Posted by: Chris McElligott at May 15, 2005 2:47 PM

>(BTW the GECKO_FORCE_PAINT_ONLOAD environment
>variable forces onload to be blocked by CSS
>background image loads.)

last I checked, that environment variable did two things: not only did it it make images block onload, it also dumped each page to a file when done loading... (http://lxr.mozilla.org/seamonkey/source/layout/base/nsDocumentViewer.cpp#1024)

Posted by: christian biesinger at May 16, 2005 1:15 PM

I know, I'm just saying that that shows how to get CSS background image loads to block onload.

Posted by: Robert O'Callahan at May 16, 2005 3:23 PM

I would like to see this used to improve search. A rendered page gives a lot of context and information than reading plain html. I can see a program which identifies large blocks of color and high contrast areas, and then OCR-ing them to get a better understanding of a website's contents, as well as relative importances.

Or how about audio browsers for the visually impaired using the generated image and knowledge of the underlying HTML to let users navigate the page "visually"? That should work even on websites that were not designed for accessibility.

Posted by: Kingsley at May 20, 2005 5:04 AM

Roc, do you know if this is in DeerPark Alpha 1?
I was thinking about taking your code and do something interesting with it...

Thanks

Posted by: Jed at June 2, 2005 11:26 AM

It is in Deer Park Alpha 1.

Posted by: Robert O'Callahan at June 2, 2005 7:37 PM

How long until spammers combine this technology with OCR, to extract email addresses that are shown as image(s) or generated by javascript? :-(

Posted by: ChrisH at June 3, 2005 12:15 AM

I'm with Chris - I'd love to see a Firefox sidebar extension that allowed Omniweb style tabs

Posted by: Jon Hicks at June 3, 2005 2:54 AM

FYI, this was broken in Deer Park Alpha 1. It should be fixed in the next nightly build. (Thanks biesi!)
the relevant bug

Posted by: Ted Mielczarek at June 3, 2005 12:52 PM

ChrisH: if you can call drawWindow, then you could just crawl the window's DOM and get the email addresses directly without OCR. There is no new issue here. (Especially because currently Web content can't use this API.)

Posted by: Robert O'Callahan at June 3, 2005 7:44 PM

Ted: thanks for that! I don't know why it worked in my build. I think it also worked in ben's build...

Posted by: Robert O'Callahan at June 3, 2005 7:45 PM

I see that you've added a drawWindow. I think it would be pretty useful to be able to draw a DOM Element as well. That would make the canvas a lot more flexible.

Is there a reason why this only works from chrome? Wouldn't a same origin check be sufficient?

Posted by: Erik Arvidsson at June 6, 2005 4:24 AM

Drawing a DOM element is a bit hard because DOM elements are not necessarily rectangular. Suppose you have a window in a paginated context or with a column, and an inline element that breaks across a page or column boundary, then the different pieces of the element could be anywhere. What do you draw in that case? Yes, it would be useful but it's hard to specify and hard to implement.

In many cases you could just clone the content into its own hidden IFRAME and render that.

Posted by: Robert O'Callahan at June 6, 2005 11:25 PM

ROC:
"
The plan is to have a .toDataURL() method that returns a data: URL containing a base64 encoded PNG.
"

In Moz/FF, is there a practical limit to the length of data: URL? I know spec says something like 2KB, but I've seen larger, and it's pertinent for Greasemonkey.

Thanks!

Posted by: Jeremy Dunck at June 15, 2005 11:31 AM

I've made data: URLs that were hundreds of kilobytes long and they worked.

Posted by: Robert O'Callahan at June 15, 2005 12:37 PM

Is there any way to implement this as a php feature in a script?

Thanks

Posted by: Crirus at June 18, 2005 3:33 AM

I'm new to this so forgive me if my question sounds incredibly dumb. Where can I find the drawWindow API/plugins and how do I install it? Can anyone point me to some general guide to do this sort of things?

I have a project that can really use this feature. Thank you in advance!

Posted by: luping at July 12, 2005 7:15 AM

Cirius: it runs on the client only.

luping: You just need one of the newest "Deer Park" Firefox builds.

Posted by: Robert O'Callahan at July 12, 2005 8:48 AM

since FireFox 1.1 shows a thumbnail of an image in the tab is it possible for me to specify an image to be used in the tab on my page? i'm not talking about the favicon image, i mean making firefox use an image that displays in the page as the tab icon. so each page will have its own icon.

Posted by: joel at July 21, 2005 6:23 AM

You can use a tag in your page to specify a per-page icon.

Posted by: Robert O'Callahan at July 21, 2005 9:22 AM