Ivan Zuzak

Experimenting with HTTP services - UrlEcho and UrlReq

2013-03-19T00:00:00-07:00

Introduction

UrlEcho and UrlReq are two HTTP services I built a few years ago when I started learning about the Web's architecture. The services are fairly generic and might be useful to other people experimenting, testing and debugging Web-based systems. They are hosted on AppEngine and free to use.

Both services receive an HTTP request which contains a definition of another HTTP message in the URL of the received request, and then do something with that message.

The UrlEcho service

Requests to the UrlEcho service should contain the definition of a HTTP response message in the URL (i.e. the status, headers and body of the response). Upon receiving the request, UrlEcho responds to the requestor with the HTTP response defined in the URL of the request. Here's an example request URL (clickable link here):

http://urlecho.appspot.com/echo?status=200&Content-Type=text%2Fhtml&body=Hello%20world!

and the corresponding HTTP response which is defined in the URL and will be echoed back to the requestor:

HTTP/1.1 200 OK
Content-Type: text/html

Hello World!

As shown in the example, the API for the service is simple (split into multiple lines for clarity):

http://urlecho.appspot.com/echo
  ?status=<HTTP_STATUS>
  &<HEADER1_NAME>=<HEADER1_VALUE>
  &<HEADER2_NAME>=<HEADER2_VALUE>
  &body=<BODY_STRING>

where the strings for header names and values, and for the body must be URL encoded.

The UrlReq service

On the other hand, requests to the UrlReq service should contain the definition of another HTTP request message in the URL (i.e. the method, url, headers and body of the request). Upon receiving the request, UrlReq performs the request defined in the URL, and forwards the received response back to the requestor. Here's an example request URL for requesting HTML rendering of Markdown input using GitHub's API (clickable link here):

http://urlreq.appspot.com/req?method=POST&url=https%3A%2F%2Fapi.github.com%2Fmarkdown%2Fraw&
Content-Type=text%2Fplain&body=**Hello**%20_World_!

the corresponding HTTP request hosted in the URL:

POST /markdown/raw 
Host: api.github.com
Content-Type: text/plain

**Hello** _World_!

and the response received from the service:

HTTP/1.1 200 OK
Content-Type: text/html;charset=utf-8
...

<p><strong>Hello</strong> <em>World</em>!</p>

As shown in the example, the API for the service is simple (split into multiple lines for clarity):

http://urlreq.appspot.com/req
  ?method=<HTTP_METHOD>
  &url=<TARGET_URL>
  &<HEADER1_NAME>=<HEADER1_VALUE>
  &<HEADER2_NAME>=<HEADER2_VALUE>
  &body=<BODY_STRING>

where the strings for header names and values, for the target URL, and for the body must be URL encoded.

URL builders and debug mode

To make the construction of such URLs easier, simple URL builder forms are available for both services: UrlEcho URL builder and UrlReq URL builder (scroll down when you get to the project page).

Also, both services have support for debug mode by appending &debugMode=1 to the end of the URLs. When in debug mode, the services return a text/plain version of the HTTP response that would be returned if the service was not in debug mode. So, it's basically a text log output of the intended HTTP response.

Why build these services?

In a way, both services change/break the default way of using HTTP -- the URL is used for more than identifying a resource and the services are just stateless shells that either echo requests or responses. They could be thought of a special kind of HTTP intermediaries.

UrlEcho permits the requestor to completely define the response it wants to receive, thus giving it the ability to "host" static HTTP resources within URLs themselves. Why is this cool/useful?

You don't need a Web server to host a simple resource -- you just construct a URL and you're set to go.
Since resources are cheap to create and throw away, you can create as many URLs as you want, when you want them.
This is especially useful for testing -- you don't want to configure many server-side resources to return hard-coded responses in order to test correct handling of that response. For example, imagine you need many simple iframes to test a JavaScript library, and you don't want to modify the server hosting the iframes each time you add or change tests. Since you already know what the responses will be, why not define them in the requests and have a simpler testing process? It makes tests easier to maintain (no need to modify the server) and understand (due to response visibility in the tests).

UrlEcho lets the requestor to wrap any HTTP request (any method, with headers and body) into a simple GET request with only an URL defined. Why is this cool/useful?

You can make any HTTP request in situations where only a simple GET is possible, or where you can only define just the URL. For example, most systems that provide Web hook callbacks only let you define the callback URL only (not the method, headers or body structure).

In general, the reasons behind using URLs to "host" HTTP messages (versus choosing headers or the body) is that it has the widest possible applicability. The URL is often the only thing you can define for a request -- e.g. in <a> and <iframe> HTML elements, and when defining Web hook callbacks. You can't set HTTP headers or the body in such situations, the URL is all you have access to. The example I gave above for UrlReq, the one for rendering Markdown text to HTML using GitHub's API -- isn't it cool how you can give a simple link to someone, that makes a POST request behind the scenes? It is.

Note also that the services don't respond just to HTTP GET requests, but also to HEAD, POST, PUT, DELETE and OPTIONS requests. This means that, for example, you can make an HTTP DELETE request to UrlReq and make it do an HTTP POST or HTTP PUT request in the background, just by playing with the URL.

Other nits and bits

The main issue with using URLs for passing data is that URLs have limited length. You just can't put 10 MB in there.

Long, ugly URLs make you go Meh.? Just use an URL shortening service on the long URL, like so: http://goo.gl/UOQPy.

A few notes about security-related issues. After successfully using UrlEcho for over a year, I noticed it stopped working for me in Chrome. Turns out, using UrlEcho for echoing back HTML documents with JavaScript scripts makes browsers think there is an XSS attack happening. For example, the following URL (clickable link here):

http://urlecho.appspot.com/echo?status=200&Content-Type=text%2Fhtml&body=%3Chtml%3E%0A%20%20%3C
head%3E%0A%20%20%3C%2Fhead%3E%0A%20%20%3Cbody%3E%0A%20%20%20%20%3Cscript%3E%0A%20%20%20%20%20%20
console.log(%22Hello%20world!%22)%3B%0A%20%20%20%20%3C%2Fscript%3E%0A%20%20%3C%2Fbody%3E%0A%3C%2F
head%3E

echoes back a simple HTML document with a console.log("Hello world!") script. However, the script will not get executed and Chrome will log the following error to the console: "Refused to execute a JavaScript script. Source code of script found within request." I suppose security hackers can find a way to trick Chrome and get around this, by tweaking the request content. But, this is something you should be aware of.

Finally, cross-origin calls to both services are enabled via CORS headers, both services are accessible over HTTPS, and the UrlEcho service responds on arbitrary subdomains, e.g. http://mysubdomain.urlecho.appspot.com/echo?....

Conclusion

More information about UrlEcho and UrlReq is available on their Web sites. Let me know if you like the services, think they're an abomination of the Web, or have any other cool ideas for HTTP services.

As for me, I really learned to appreciate nice and simple HTTP services with generic applicability. There is a nice list of other such services in my blog post The Web Engineer's online toolbox.

Google Scholar pointed me to a plagiarism of my paper

2012-12-03T00:00:00-08:00

I love Google Scholar, especially the recently introduced Updates feature. In a nutshell, this feature continuously searches for and recommends new articles relevant to your research based on a statistical model that incorporates your publications and citations. And it works really well! Google Scholar already recommended me dozens of relevant papers and, because Google keeps a good watch over the Web, it recommends papers soon after they are put online.

However, two weeks ago something special happened. As I opened and glanced one of the papers recommended by Google Scholar, I realized that the recommended paper was a plagiarism of one of the papers I co-authored and was published two and a half years ago. Yes, I'm 100% certain that the paper is a plagiarism; 80%-90% of the text and 100% figures were just copy/pasted. Only the abstract and title of the paper have been significantly changed since the authors probably thought this was the only way to detect foul play (since the title and abstract are usually the only parts of papers that are available for free and indexed by search engines). The original paper I co-authored is here (link to publisher site with the paper is here) while the plagiarized paper is here (link to publisher site with the paper is here.

I don't want to turn this blog post into a long rant about honesty, but I do want to highlight two points. First, without going into who plagiarized the paper, where they work or live, and in which journal the plagiarism was published -- this is just another piece of evidence of the corruption and apathy that's more and more present in all segments of academia and higher education.

Second, Google Scholar kicks ass! I've made no effort whatsoever in finding this plagiarism, it was all Google Scholar. While the service didn't really tell me that it was a plagiarism of my paper (just that it was relevant material), that's just a tiny step from here. Imagine what other things the service could do if Google made a stronger effort. Imagine publishers partnering with Google to help them find plagiarisms. Or imagine Google creating an open, Web-based platform for academic publishing and relevance/impact computation. (Google, I know you're reading this. If you need help -- call me. You know my number.) Anyway, good things are on their way, I'm sure of it. That's why I'm happy to see people boycotting academic publishers that lock-in research papers behind paywalls. And that's why I'm glad that companies like Coursera, Udacity and Khan Academy and services like Google Scholar are transforming university-level education and shaking up everyone in academia.

We reported everything to the publisher of the journal which contains the plagiarism, but haven't received any real response for more than 12 days. If anyone has been in a similar situation - let me know if you pursued the issue in other ways and how that turned out (e.g. through institutions or legal suits).

The Web engineer's online toolbox

2012-11-18T00:00:00-08:00

I wanted to compile a list of online, Web-based tools that Web engineers can use for their work in development, testing, debugging and documentation. The requirements for a tool to make the list are:

must be a live Web application (no extensions or apps you have to host yourself),
free to use (some kind of free plan available),
generic applicability (not usable only for a specific application/platform),
and must be useful to Web engineers (not just for Web site design)

The current version of the list is shown below and is based on tools which I use or have used. Two of my personal favorites are RequestBin and Hurl which I use regularly for almost every project. A few of the tools I haven't used in real projects but just played with them and thought they were really cool.

Since I've probably forgotten some excellent tools, and because new tools will be made in the future - the list will be a "live list", updated as the set of available tools changes. I didn't categorize the tools in the list since many tools may be used in many phases of software development.

Want to suggest a new tool or help me write better short descriptions of the tools? Add a comment below or make a pull request.

The Web engineer's online toolbox (last edited on May 18 2013)

RequestBin
- Lets you create a URL that will collect requests made to it, then let you inspect them in a human-friendly way.
Hurl
- Makes HTTP requests. Enter a URL, set some headers, view the response, then share it with others.
- Similar tools: REST test test, Apigee console, Web-Sniffer.
Runscope
- A proxy for inspecting API calls, providing several features of Hurl and RequestBin.
httpbin
- A HTTP request & response service that covers all kinds of HTTP scenarios (e.g. different HTTP verbs, status codes and redirects).
- Similar tools: Hang, UrlEcho, Mocky.
REDbot
- A robot that checks HTTP resources to see how they'll behave, pointing out common problems and suggesting improvements.
- Similar tools: HTTP lint.
WebGun
- API for creating templated webhooks.
- Similar tools: UrlReq.
Webscript
- Webscripts are short and hosted scripts, written in Lua. They can respond to HTTP requests or run as cron jobs.
ClickHooks
- A URL shortener service that provides callbacks via HTTP POST when people go to a shortened link.
MailHooks
- Lets you receive emails via HTTP POST (aka webhooks). You can create many hooks that will give you an e-mail address that, when e-mailed, parses and POSTs the message to the URL you specify.
Quilla
- Provides short-links where people can contact you, where each submission on your short-link will send you an email. A kind of a HTTP->SMTP proxy.
Apify
- Exposes datasets locked within HTML documents for which there are no APIs. APIfy extracts data from structured markups and converts it to JSON APIs.
Unicorn
- W3C's unified validator, which performs a variety of checks from the popular HTML and CSS validators, as well as other useful services.
- Similar tools: HTML lint.
HTML tidy
- HTML Tidy is a tool for checking and cleaning up HTML source files.
Feed validator
- W3C's validator for RSS and ATOM syndicated feeds.
JSONLint
- JSON validator.
Link checker
- Extracts links (recursively) from a Web site and checks that no link is defined twice, that all the links are dereferenceable and warns about HTTP redirects.
Host tracker
- Website monitoring service with distributed ping/trace check, periodic monitoring, email/sms/IM notifications and statistics.
- Similar tools: Down for everyone or just me, Pingdom ping service, IsItUp, Ping brigade.
ViewDNS
- Collection of DNS and network-level tools, such as reverse IP lookup, DNS record lookup and traceroute.
Necrohost
- List of URLs that simulate a variety of network connectivity issues e.g. slow response, unresolvable DNS and 404.
Mirrorrr
- Application that mirrors the content of URLs you supply. Rewrites the fetched page to mirror all content, including images, Flash, Javascript, CSS, and even favicons.
SSL Checker
- Test SSL certificates of online sites and helps with identifying any problems.
CSR/Cert decoder
- Decode and check your CSRs and SSL certificates.
SSL Server Test
- Performs a deep analysis of the configuration of any SSL web server on the public Internet.
Loadzen
- Cloud-powered load testing service.
Pingdom Full page test
- Enables users to test the load time of a page, analyze it, monitor, find bottlenecks and export results in HAR format.
- Similar tools: Web page test.
Google PageSpeed Insights
- Analyzes the content of a web page, then generates suggestions to make that page faster.
HAR viewer
- Visualizes HTTP Archive (HAR) log files created by HTTP tracking tools.
PCAP Web Performance Analyzer
- Converts a PCAP file to HAR, provides a HTTP waterfall view using HarViewer and Page Speed suggestions for your network trace.
CORS proxy
- Allows JavaScript code on your site to access resources on other domains that would normally be blocked due to the same-origin policy.
Browserling
- Interactive cross-browser testing using all major browsers and versions. Browsers within your browser.
WebSocket Echo Test
- Test a WebSocket connection from your browser against our WebSocket echo server.
YQL
- An expressive SQL-like language that lets you query, filter, and join data across Web services.
Webshell
- An API which allow accessing any data on the programmable Web using APIs.
Yahoo Pipes
- A graphical user interface for building data mashups that aggregate web feeds, web pages, and other services.
Apiary
- Language and tool for generating REST API documentation with an interactive inspector.
- Similar tools: Swagger.
JSFiddle
- A playground for Web site developers: code editor and repository for snippets built from HTML, CSS and JavaScript.
- Similar tools: JSBin
Google Feed API
- Enables looking up feed URLs for a site (example), searching for feeds by keywords (example) and returning a JSON representation of a feed (example).
About My Browser
- Extracts and displays browser info (browser, version, OS, cookies, JS, screen resolution, etc) and stores this info as a shareable Web page.
- Similar tools: http://browserspy.dk/, SupportDetails

Forgotten TODOs: ideas for contributing to open-source projects

2012-03-03T00:00:00-08:00

EDIT (March 08 2012): This post got a lot of feedback on Hacker News and Reddit. I'd like to thank all the people on their positive and constructive thoughts and address some of the feedback.

First, there was some really good advice on advanced usage of TODOs, such as using tools for tracking them systematically and using different TODO tags (such as TODO, FIXME, TBD, and XXX) to indicate the relevance/urgency of the TODO. Since a lot of people mentioned that they also use FIXMEs, I've also included their count in the table in the post.

However, the comments I liked the most were the ones that suggested other generic ideas for contributing to open-source projects, especially the ones that mentioned writing documentation and unit tests. I can't stress enough how good advice this is.

There's also been some negative feedback, mainly focused on explaining that newcomers should not target TODOs because they are usually either very hard to fix, will have little value if fixed, or are not meant to be fixed by someone who is not the project maintainer. In addition, some comments say that newcomers should focus on fixing bugs that they have experienced themselves and implementing features that mean something to them ("scratch your own itch"). While I agree with these points and have tried to make them clearer in the post, I have to stress that the main point of this post is to help people get ideas when they don't have ideas of their own. Therefore, regardless of their programming skills and the impact of fixing a TODO, I do think that looking through TODOs will help in generating ideas.

I often talk to students that want to contribute to open-source projects, but just don't have an idea what to work on. Here's a tip if you're in a similar situation (e.g. you want to apply for GSOC) :

1 git clone repository_url_of_some_open_source_project target_directory
2 grep -RIn TODO target_directory/*

So, find the URL of the repository project you want to contribute to, checkout the repository using git/mercurial/svn and then find all the TODOs in the source code using grep. The -RIn flags will tell grep to do a recursive search (-R), skip binary files (-I) and include line numbers (-n) for results. And that's it! Go through the results and pick whatever you find interesting and within your skill range.

Most open-source projects have issue trackers for organizing bug reports and feature requests that need to be worked on. However, some projects don't use issue trackers, and there's an important difference between issues in these trackers and what you'll find by digging through the source code. Issues are commonly a result of using a system or library, they are created by users, and focus on bugs and features. TODOs on the other hand are a result of system development and are created by developers. Software developers often make notes (TODOs) of things that should be fixed or could be done in a better way, but don't have the time to do it right away.

These notes are focused more on improving the internal operation of the system and code quality, rather than on externally observable features, and include anything from checking specific algorithm edge-cases to larger refactoring ideas. And as time goes by, these TODOs are often forgotten since they are not systematically tracked and more effort is put into developing new features and fixing bugs. So, if you really want to get your hands dirty, TODOs are a good place to start looking for work. At first, project owners and maintainers will often be cautious about letting you hack into system internals and refactor, but will generally be very grateful if you really want to fix those forgotten TODOs. In a way, it's often a bit harder to start working on a TODO than on an tracked issue.

Here's the number of TODOs and FIXMEs for the top 15 watched projects on GitHub:

Project name	Number of TODOs + FIXMEs
bootstrap	7+0
nodejs	904+278 (many of these are v8 TODOs)
rails	77+55
jquery	7+0
html5-boilerplate	2+0
homebrew	22+6
spoon-knife	0+0
impress.js	0+0
backbone	4+0
diaspora	16+0
three20	25+0
devise	2+0
jquery-mobile	60+0
three.js	43+6
express	3+0

Of course, in some projects you won't find any TODOs, in others you'll find false positives and TODOs from 3rd party libraries, but in general - forgotten TODOs are a cool way of finding ideas for contributing to open-source projects.

Back to blogging

2012-02-24T00:00:00-08:00

Forgive me, InterWebs, for it has been a year since my last blog post. 2011 was a very busy+wonderful year for me, and I've had the opportunity to meet many nice people, visit many cool places and work on many interesting projects.

I gave a talk at the Knowledge Media Institute of the Open University, UK about inter-widget communication based on the work that Marko and I did in this field. I've had the pleasure of meeting Fridolin Wild, who organized the talk, and his research group at the KMI. I also published two papers on inter-widget communication and the pmrpc library.
I published two papers on the FSM-based formalism for modeling RESTful systems and presented one of the papers at the ICWE conference (held in Paphos, Cyprus). I've also had the pleasure of meeting Silvia Schreier, discussing many REST-related ideas with her and starting work on Feral (a framework for developing RESTful systems).
For the Programming Language Translation course at the University of Zagreb (where I work as a teaching assistant), I've implemented an assembler and simulator for a simple RISC processor, in pure JavaScript, which can be used in Web browsers and in NodeJS apps.
I also played with randomness analyses for two lotto systems in Croatia, but since statistics isn't my strongest side, I'm quite far from interesting results. Something similar to the statistics for UK lotteries. And yes, I do know that lotto really is random.
Finally, I'm nearing the deadline for finishing my Ph.D. I'm nearly done with my last seminar, and I'm working on my thesis and thinking about what the future will bring.
This Week In REST is still alive and popular, but I update it less frequently due to lack of time.

So, over the next few months I'll (hopefully) be writing blog posts about some of these things, especially REST. I've already updated a few of my old posts that continuously generate interest.

GitHub hosted comments for GitHub hosted blogs

2011-02-18T00:00:00-08:00

EDIT (January 25 2011): A while back, the GitHub team has updated the GitHub API to v3 which is now even more cool. Since the new API supports fetching both the Markdown source and the HTML rendering of a comment, this eliminates the need for rendering the comments on the client-side using the Showdown library as it was done before. I've updated the blog post to reflect this and some other minor API changes.

Recently switching from a Wordpress.com-hosted blog to a GitHub hosted blog made me curious if blog comments could also be somehow hosted on GitHub. Why host comments on GitHub? Other than it being cool, open and hackable, it would enable that all content for the blog be hosted on GitHub.

So, here's what I wanted from the future commenting system:

must be hosted completely on GitHub.
must be tied to the GitHub repository which hosts the blog.
must provide user login, tie each comment with the user who made the comment, and give both blog owners and users control over their comments.
must be possible to pull the comments for a post from the system and show them on the post's Web page.
should be easy to set-up by the blog owner and easy to use by blog readers.

These requirements ruled out using Gists, asking users to fork the repository and submit pull-request to a special comments file (heh, this was actually the first thing that I thought of) or using any external service for comment creation and/or hosting.

The solution I came up with is to use the GitHub issue tracker for creating and hosting comments and the GitHub API for pulling comments from the issues into the blog's Web page ~~and GitHub Flavored Markdown for rendering comments within the Web page using JavaScript~~. Here's how all of this works.

First, for each blog post you plan to publish, create an issue in your blog's GitHub repository issue tracker. For example, here's the issue for this blog posts. Notice that all issues have a common base URL (https://github.com/izuzak/izuzak.github.com/issues/ in my example) and a unique id (12 in my example). Readers of your blog will use the issue tracker if they want to leave a comment.

Second, add a link to each of your blog posts pointing to the Web page of the issue you created for comments. Since, you're probably using Jekyll to generate your blog, a simple way of doing this is to add the above mentioned id of the issue to the YAML front matter block of each blog post and then add a Liquid template in your layout file to retrieve the id and generate the full issue URL. For example, here's the YAML front matter for the blog post your reading (commentIssueId is the id of the post's issue in the repository Issue tracker):

1 ---
2 layout: post
3 title: GitHub hosted comments for GitHub hosted blogs
4 tags: github, blog, comments, hosting
5 commentIssueId: 12
6 ---

And here's the snippet of the layout file for generating blog posts (page.commentIssueId part surrounded with curly braces is the Liquid template which pulls the commentIssueId of the post):

1 <div id="comments">
2   <h2>Comments</h2>
3   <div id="header">
4     Want to leave a comment? Visit <a href="https://github.com/izuzak/izuzak.github.com/issues/{{page.commentIssueId}}"> this post's issue page on GitHub</a> (you'll need a GitHub account. What? Like you already don't have one?!).
5   </div>
6 </div>

Third, add a JavaScript script to each blog posts which pulls the post's comments from the issue using the GitHub API. The GitHub issues API is exactly what we need here - make an ~~JSONP~~ XHR request to the API endpoint and you'll receive all the comments for a specific issue. For each comment you get the GitHub user id and gravatar id of the person who made the comment, the comment id in the issue, created and updated timestamps and the body of the comment. Again, since you're probably using Jekyll, it's easiest if you put the code in the layout file for blog posts. For example, here's the code for pulling in the comments for this blog post, using jQuery (notice again that you need a Liquid template to specify the URL of the issue for which you want to pull the comments):

 1 <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js"></script>
 2 
 3 <script type="text/javascript">
 4   function loadComments(data) {
 5    // ...
 6   }
 7 
 8   $.ajax("https://api.github.com/repos/izuzak/izuzak.github.com/issues/12/comments", {
 9     headers: {Accept: "application/vnd.github.full+json"},
10     success: function(msg){
11       loadComments(msg);
12    }
13   });
14 </script>

But wait, doesn't the same-origin policy (SOP) apply when making an XHR request to the GitHub API endpoint from the blog's page? Yes, yes it does. And until recently, the only way to get around SOP for accessing the GitHub API was to use JSONP. However, using JSONP disables the possibility of setting HTTP request headers which we need to correctly set the desired API response media type via the Accept header (see code above). However, as of recently, GitHub supports the CORS specification which enables cross-origin requests. The way in which GitHub enables CORS for its API is by requiring API users to register the Web sites that access the API as OAuth applications. Therefore, in order to enable CORS for a GitHub-hosted blog, you need to go to the GitHub form for registering OAuth applications and create a new application by specifying your blog's URL in the Main URL field and the Callback URL field. Notice that this approach works (and is the same) both for blogs on GitHub subdomains (e.g. izuzak.github.com) and for blogs hosted on GitHub that have a custom domain (e.g. http://ivanzuzak.info).

Fourth, insert the pulled comment data into the blog post's HTML and render the body of the comments using GitHub Flavored Markdown. Inserting the comment data into the DOM is easy once you decide which data you want to display. However, since issue comments may be written in GitHub Flavored Markdown, some JavaScrip code is needed to translate the comment to HTML. Fortunately, GitHub has a JavaScript library for that also - a modified version of the Showdown library for converting Markdown into HTML. Since the pulled data contains a HTML rendering of the comment's Markdown source, there's no need to do any processing of the comment text. Again, it's easiest if you put the code in the layout file for blog posts. For example, here's the code which I use for inserting comments:

 1 <script type="text/javascript" src="http://datejs.googlecode.com/svn/trunk/build/date-en-US.js"></script>
 2 
 3 <script type="text/javascript">
 4 
 5   function loadComments(data) {
 6     for (var i=0; i<data.length; i++) {
 7       var cuser = data[i].user.login;
 8       var cuserlink = "https://www.github.com/" + data[i].user.login;
 9       var clink = "https://github.com/izuzak/izuzak.github.com/issues/12#issuecomment-" + data[i].url.substring(data[i].url.lastIndexOf("/")+1);
10       var cbody = data[i].body_html;
11       var cavatarlink = data[i].user.avatar_url;
12       var cdate = Date.parse(data[i].created_at).toString("yyyy-MM-dd HH:mm:ss");
13 
14       $("#comments").append("<div class='comment'><div class='commentheader'><div class='commentgravatar'>" + '<img src="' + cavatarlink + '" alt="" width="20" height="20">' + "</div><a class='commentuser' href=\""+ cuserlink + "\">" + cuser + "</a><a class='commentdate' href=\"" + clink + "\">" + cdate + "</a></div><div class='commentbody'>" + cbody + "</div></div>");
15     }
16   }
17 </script>

Also notice that I use the DateJS library to pretty-print the comment dates and that I fetch the Gravatar images of users who made the comments.

Finally, add CSS rules to make the comments look pretty. And if you're extra geeky and love GitHub like me, make the comments look like GitHub code comments:

And that's it! Creating the system was really easy - GitHub already had all the pieces necessary (Issue tracker, API, OAuth application registration form~~, Markdown JavaScript library~~) and all I needed was a bit of glue to tie all of them together. I'm using this as the commenting system for my blog until some serious problem turns up. I like how the system integrates with GitHub login, how all blog comments are visible on the issue page and the overall simplicity of using the system. What I didn't find an easy way of doing is creating issue comments directly from the a blog post's Web page, without making the user navigate to the Issue tracker page (really hard to do without an external service since the blog is a static site). Other than that, I'm not sure what will happen when spam bots start invading GitHub issue trackers.

You can see the complete code I'm using for the commenting system in the following files: general layout for blog posts and CSS rules. Let me know if you find a better system for hosting blog comments on GitHub or improving this one. @izuzak

Enabling PubSubHubBub for GitHub hosted blogs

2011-01-02T00:00:00-08:00

EDIT (August 20 2012): Sergey Lukin has been doing some digging and discovered two very interesting things which make some of the claims in my post no longer valid.

First, GitHub now allows URL query parameters in post-commit Web hooks. Although this isn't mentioned in the official docs, it just works. In other words, you can specify a Web hook URL like http://your.server.com/web_hook_resource?param1=value1&param2=value2 and GitHub will make a POST request to the full URL, without discarding the query parameters, as before.

Second, the reference PubSubHubBub hub at http://pubsubhubbub.appspot.com doesn't strictly follow the PSHB specification with regard to the parameters passed in the new content notification HTTP POST request. Specifically, the spec says that the hub will read the notification parameters from the HTTP request body. However, the notification request will be correctly processed even if the parameters are specified as URL query parameters, not in the message body. The reason why this works is that the reference hub is hosted on Google's AppEngine and implemented using the webapp2 Python framework. This framework parses parameters both from the message body and URL query part and exposes them as a single object, making it irrelevant how a parameter was actually passed.

The result of these two discoveries is that you no longer need to use a proxy server to ping a PSHB hub using GitHub post-commit hooks, as described in the blog post below. You can just use a direct Web Hook to the reference hub. For example, I would specify this URL as the Web Hook to ping the hub when my blog's ATOM feed is updated: http://pubsubhubbub.appspot.com/publish?hub.mode=publish&hub.url=http%3A%2F%2Fivanzuzak.info%2Fatom.xml. Just replace the value of the hub.url parameter with your blog's ATOM/RSS feed URL and that's it!

However, again, notice that this works only because the reference PSHB hub is not strictly following the PSHB specification. If you are using another PSHB hub, check that it can accept parameters passed in the URL query part.

Great work, Sergey!!

I've recently switched from a Wordpress.com hosted blog to a GitHub hosted blog on a custom domain using GitHub Pages. And one of the first things I wanted to enable for the new blog was PubSubHubBub (PSHB) support. Generally, in order to enable PSHB for a blog, two things need to be done:

1) add a <link rel="hub" href="http://url.of.a.pshb.hub" /> to the blog's ATOM feed, linking to a PSHB hub, e.g. the reference hub running on appengine http://pubsubhubbub.appspot.com (see PSHB specification Section 5. - Discovery)

2) every time a new blog post is created or an old blog post is updated - notify the hub from step 1) by making a HTTP POST request with the body containing the blog's ATOM feed URL (see PSHB specification Section 7.1. - New Content Notification)

For GitHub hosted blogs, step 1) isn't a problem as it only requires you to insert one line of text into your blog's ATOM XML feed. However, step 2) is a problem. The good news is that each GitHub Pages site is also a GitHub repository and GitHub supports service hooks, predefined services which GitHub notifies each time the repository is pushed to. The bad news is that, although many services like Twitter and Campfire are supported, PubSubHubBub hubs are not supported. So basically, you need to make a custom service. Now, a special type of service hooks are generic post-receive hooks, user-specified URLs (also called webhooks) to which GitHub makes a HTTP POST request each time the repository is pushed to. And now for more bad news. First, HTTP POST requests made to post-receive hook URLs have a predefined and non-changeable message body. Therefore, because a PSHB notify request must contain certain parameters in the body of the HTTP POST request, there is no way of specifying these parameters since the body is predefined by GitHub. Second, requests made to post-receive hook URLs are stripped from the URL query part. Therefore, even if PSHB supported passing notification parameters through the query part of the request URL, GitHub wouldn't allow that either. I've asked about this issue on the GitHub forum and got a semi-satisfying response.

Until there's a better solution, here's an ugly-hack solution. I created an AppEngine proxy service, to be used as a post-receive hook, which makes PSHB notifications only to the reference PSHB hub when a POST request is made to the service. Since GitHub doesn't allow query parameters in the URL, the proxy service will expect the blog's ATOM feed URL to be passed as the suffix of the path part of the URL (e.g. instead of making a request to http://www.service.com?blogUrl=http://some.url.com, a request would be made to http://www.service.com/http://some.url.com).

So, here's a step-by-step guide for enabling PubSubHubBub for your GitHub hosted blog:

1) Add the following lines to your blog's ATOM feed XML:

<link href="http://url.of.your.blogs.atom.feed.xml" rel="self"/>
<link href="http://pubsubhubbub.appspot.com/" rel="hub"/>

where http://url.of.your.blogs.atom.feed.xml is what it says it is - the URL of your blog's ATOM feed (e.g. mine is http://ivanzuzak.info/atom.xml). If this line is already present in your blog's ATOM feed, you don't have to add it again. Naturally, you will perform this step by editing your atom.xml source file and push it to the GitHub repository hosting your blog. You can see my atom.xml source here.

2) Open the admin page for the GitHub project hosting your blog and navigate to the Service hooks tab. Now, add a new post-receive hook with the following URL:

http://urlreq.appspot.com/pshbpinggae/http://url.of.your.blogs.atom.feed.xml

where, urlreq.appspot.com is the AppEngine proxy service I mentioned earlier and, again, http://url.of.your.blogs.atom.feed.xml is what it says it is - the URL of your blog's ATOM feed. For example, the post-receive hook URL for my blog is http://urlreq.appspot.com/pshbpinggae/http://ivanzuzak.info/atom.xml. Passing parameters this way is really ugly, but currently there's no better way of doing it.

Now every time you push a new blog post to the blog's GitHub repository, GitHub will make a POST request to the urlreq AppEngine proxy service, the service will notify the reference PSHB hub, the hub will fetch your blog's ATOM feed and notify all subscribers of the new feed entry. And if you make a commit the the repository which doesn't contain a new blog post, the service will still notify the hub, but the hub is smart enough to detect that no new entries are present in the ATOM feed so no subscribers will be notified.

GitHub pages are really cool for hosting blogs and GitHub should do more in supporting them - it would be cool if a PubSubHubBub service hook was added to the list of Service hooks or if generic post-receive hooks supported specifying custom parameters for the hooks (either through the URL query part or through the HTTP request body). @izuzak

Pmrpc discovery and publish-subscribe support + systematization of cross-context browser communication systems

2010-06-15T00:00:00-07:00

EDIT 29 August 2011: I've published two papers on concepts presented in this blog post. First, I've published a paper about the Pmrpc library, its architecture and benefits over other libraries, at the MIPRO 2011 conference. Also, I've published a paper about the classification framework for cross-context communication in Web browsers. The paper presents a more detailed and advanced analysis of cross-context communication than the one addressed in the second part of this blog post. Furthermore, a few moths ago I gave a talk on inter-widget communication at the Open University, UK. Since inter-widget communication significantly overlaps with cross-context communication, the talk touches on many concepts from the cross-context communication framework. Links for downloading the papers and presentation slides are in the talks and papers section.

The pmrpc cross-context browser communication library has grown from the last time I blogged about it. Last time I wrote about adding support for RPC-style communication for WebWorkers. Today I'll first introduce two new features - dynamic discovery of remote procedures and a publish-subscribe communication model. Also, I'll write about our systematization of existing systems for cross-context browser communication.

Discovery and publish subscribe

Up until now, pmrpc supported only remote procedure call as the communication model for inter-window and WebWorker communication. This model of communication is based on a client-server interaction where the client both a) has an object reference of the destination window/webworker context and b) knows the name of the remote procedure it wants to call. Here's an example of a pmrpc RPC call:

1 // server [window object A] exposes a procedure in it's window
2 pmrpc.register( {
3   publicProcedureName : "HelloPMRPC",
4   procedure : function(printParam) { alert(printParam); }
5 } );

1 // client [window object B] calls the exposed procedure remotely, from its own window
2 pmrpc.call( {
3   destination : window.frames["myIframe"],
4   publicProcedureName : "HelloPMRPC",
5   params : ["Hello World!"]
6 } );

This works well for cases when the client knows how to obtain a reference of the server context object. However, it turns out that in many real-world cases the client does not know which window/iframe/worker implements the procedure it wants to call. Mashups and widget portals are the best example. Imagine that both the client and the server are widgets on iGoogle. Although you may be in control of both the client and server iframe contents, in most cases you will not know the names of iframe elements which contain them and which are located on the top container page. And you need the values of the name attributes so that you can obtain a reference to the iframe object which you could then use to call postMessage.

This referencing problem was the motivation for implementing both the discovery and publish-subscribe features. Both features enable clients to call a remote procedure without a-priori knowing which destination object implements the procedure. First, the publish-subscribe feature enables clients to perform a remote call without specifying the destination context object reference. If a publish-subscribe call is made, the pmrpc infrastructure will make a RPC call for the specified procedure to all window, iframe and WebWorker object it can find (thus simulating a "publish" event on a channel named by the remote procedure name). The API for using the publish-subscribe feature is simple; it reuses the pmrpc.call method and instead of passing the reference of the destination context object, a "publish" string parameter is passed instead:

1 // server [window object A] exposes a procedure in it's window
2 pmrpc.register( {
3   publicProcedureName : "HelloPMRPC",
4   procedure : function(printParam) { alert(printParam); }
5 } );

1 // client [window object B] calls the exposed procedure remotely, from its own window
2 pmrpc.call( {
3   destination : "publish",
4   publicProcedureName : "HelloPMRPC",
5   params : ["Hello World!"]
6 } );

Under the hood, the publish-subscribe feature uses the discovery feature to discover all window, iframe and WebWorker context objects and retrieves their references. Also, the publish-subscribe feature may be used with other pmrpc features, like access control and retries.

The discovery feature enables dynamic discovery of procedures registered at remote contexts and their filtering using various optional criteria. Discovered procedures may be filtered by the destination window/iframe/webworker context object, destination context origin (specified as a regular expression) and destination procedure name (also specified as a regular expression). For each discovered procedure that wasn't filtered out, the procedure name, origin of the destination context, the access control rules specified for the procedure and the destination context object. The feature is implemented as a new pmrpc method - pmrpc.discover. The method is asynchronous and accepts a parameter object which specifies the filters and a callback method which will be called when all registered procedures are discovered. Here's an example:

1 // server [window object A] exposes a procedure in it's window
2 pmrpc.register( {
3   publicProcedureName : "HelloPMRPC",
4   procedure : function(printParam) { alert(printParam); }
5 } );

 1 // client [window object B] discovers and calls the exposed procedure remotely, from its own window
 2 var discoCallback = function(discoveredProcedures) {
 3   if (discoveredProcedures.length > 0) {
 4     console.log(discoveredProcedures[0].publicProcedureName);
 5     console.log(discoveredProcedures[0].destinationOrigin);
 6     console.log(discoveredProcedures[0].procedureACL);
 7     pmrpc.call( {
 8       destination : discoveredProcedures[0].destination,
 9       publicProcedureName : "HelloPMRPC",
10       params : ["Hello World!"]
11     } );
12   }
13 };
14 pmrpc.discover( {
15   destination : [window.frames["someiFrameName"], window.frames["myIframe"]],
16   origin : ".*goodOrigin.*",
17   publicProcedureName : ".*Hello.*",
18   callback : discoCallback
19 } );

The implementation of the discover method is based on recursively visiting every iframe in the window.frames object starting from window.top:

The method fetches a reference to the top window object (window.top), adds the reference to the result array and repeats the procedure for each nested iframe (by iterating through the window.frames object). This way, all iframes in the current window are visited. This process is depicted by the figure below.
The method adds all local web worker objects to the result array.
On each context object in the result array, a special pmrpc infrastructure method is invoked to obtain a list of registered procedures.
The list of all obtained procedures is filtered by the specified criteria.
The method invokes the callback, passing the list of discovered procedures as a parameter.

It's not an ideal way of discovering remote contexts, but it's better than nothing and works with the latest versions of all major browsers.

Systematization of cross-context browser communication systems

It's evident that the WWW is becoming ever more componentized as almost every web site is a mashup, contains widgets or uses WebWorkers. As a result, there has been a increase in development of systems that enable communication between these different browser contexts - windows, iframes and WebWorkers. Pmrpc is only one example of such system and lots of other systems exist. However, as the number of these systems increases, the understanding of each systems' capabilities and their comparison and evaluation becomes a more difficult task since no effort is being made to systematize this ecosystem.

This is why Marko and I are trying to create a systematization of existing cross-context communication systems. Our work on this systematization is in its early stages but is openly available to anyone at a pmrpc wiki page. At the moment we are trying to do the following:

Make a list of all mechanisms for cross-context communication. We are including both in-browser mechanisms like the postMessage API, cookies and URL fragment, and 3rd party libraries like easyXDM and pmrpc.
Define dimensions for classifying the gathered systems. Each dimension defines a system characteristic whereas values along each dimension define a set of possible design choices. Currently, we have defined about a dozen dimension and are still thinking some through. Here are some examples: Type of system (browser built-in, pure client-side framework, server-mediated initialization framework, server-mediated communication framework), Transport mechanism (cookies, URL fragment, postMessage, ...), Programming model (message-oriented, shared memory, RPC, publish-subscribe).
Evaluate the listed systems according to the defined dimensions.

I'd love to hear what you think of the new pmrpc features and the systematization (see more examples at our API docs page)! Do you know of any cross-context communication systems that are not listed? Do you think some other system characteristic should be included in the classification? @izuzak

Why understanding REST is hard and what we should do about it - systematization, models and terminology for REST

2010-04-03T00:00:00-07:00

EDIT 29 August 2011: I've published a paper on the FSM-based formalism for describing RESTful systems at the ICWE 2011 conference. The formalism presented in the paper is a more detailed and advanced version of the one presented in the second part of this blog post. Links for downloading the paper and presentation slides are in the talks and papers section.

EDIT 25 January 2012: I've published another paper on the FSM-based formalism for RESTful systems, this time in the Journal of Web Engineering. This paper is again a revised and extended version of the previous paper. Furthermore, the paper explores the practical challenges and benefits of using the presented formalism. Again, for more details see the talks and papers section.

This is going to be another long post, so I'm using the introduction as an overview again.

Introduction

This post is about understanding REST, the software architectural style behind the World Wide Web. My Ph.D. research, which I'll write about some other time, pushed me on the road of REST and over the last year I've been reading lots of research papers, lots of blogs, lots of mailing lists, lots of tweets, lots of videos, wikis, books and IRC transcripts on REST and I've also recently started the This Week in REST wiki and blog. In other words, I've read almost everything I could find on REST. So, in the first part of this post I'll write about several thoughts which stuck with me while researching REST:

REST is and will continue to be important - it's the foundation of the WWW and will be the foundation of its future stages and dimensions, like the Semantic Web and the Web of Things.
Understanding REST is hard - the material on REST is fragmented and there is no clearly defined and systematized terminology or formal models used in discussions.
We can and should fix the problem - there are enough motivated and smart people who can, through an open and collaborative process, create a better way of fully understanding REST.

In the second part of the post I'll add to the pile of scattered fragments on understanding RESTful systems and describe an abstract model of a simplified REST user-agent using finite state machines. It's a very basic model but serves the purpose of showing that it is both possible and useful to develop such formal models.

I hope this post will motivate people involved and interested in REST to contribute to a process for improving the understanding of REST. I was planing to write my thoughts as a paper for the First International Workshop on RESTful Design (WS-REST 2010) but didn't have enough time. Nevertheless, I hope that people who are going to the workshop will talk about this issue. I'll start a thread on the REST mailing list for following up on these ideas, so please comment there if you want to join the discussion (EDIT: thread is here).

Representational State Transfer and why it's important

Although this post is about understanding the REpresentational State Transfer (REST) software architectural style, I'm not going to write another REST introduction. If you don't know anything about REST, start from the wikipedia article and go from there.

What's more important for this post is that REST is and will continue to be an essential part of the WWW. First, through the Web's core technologies, REST is responsible for most of the good properties of the current Web as a large-scale network-based system. Second, REST principles guide the development of both the next "major" generations of the Web, like the Semantic Web, the Web of Things and the Real-time Web, and the "minor" incremental changes of continuous evolution. Whereas the Semantic Web is about exposing and interlinking data on the WWW, the Web of Things is about connecting and exposing every physical thing to it and the Real-time Web is about real-time access to that data and things. Therefore, understanding the Web today, it's future evolution and the whole technology jungle on top is based on the understanding of REST. Third, REST is important as a research subject on its own (although some people think it isn't EDIT: Subbu elaborated on this misunderstanding below in the comment section) and will be analyzed and built upon to define new architectural styles.

Understanding REST

In my experience, understanding REST is far from easy. Of course, not everyone needs to understand REST and different people may want or need to understand REST at different levels and depths. However, despite it being a highly technical and academic concept not everyone should easily and fully grasp, I think it's hard even for people aimed at - computer scientists, software engineers and web architects. After reading everything I could find on REST and the WWW, here are the problems I believe are responsible for this:

First, the REST master reference, Roy Fielding's doctoral dissertation (and subsequent academic papers), is not completely suitable for fully understanding REST. For a doctoral dissertation, it's severely lacks images explaining important concepts in the two key sections on REST. However, I bet everyone knows the single image in those sections almost by heart from how much it's being repeated everywhere (for those who don't, that's the image on the right). The same could be said for the lack of (formal) models to describe specific properties, elements or views of REST, like HATEOAS. It's as if the very clean and systematic approach of explaining software architecture and network-based architectural styles present in the first few chapters of his Ph.D. somehow vanished in those two sections. I am aware that there are no formal models for defining architectural styles as a whole (at least, there weren't any then), but using models like state machines, Petri nets or process calculi to describe and explain parts of it would have definitely been of great benefit. It's that absolute clarity of formal models that helps when you need to have a complete grasp of a concept. Furthermore, since models are abstract methods for explaining specific problems, people only need to research the underlying model after which the specific use of the model is unambiguously clear. And just to be clear, I think Roy's dissertation is overall well written and has had a big impact on both the academia and industry, and I very well know that making everyone on the planet happy with a research paper is idiotic (looking for a good Ph.D. comics reference for this), but I still wish those two sections were a bit more thorough and polished.

Second, since Roy's dissertation doesn't have clear answers for all questions, people start discussions over understanding specific parts of REST all over the Web. Relevant discussions are scattered over many mailing lists, blogs, Twitter accounts, wikis, academic papers, videos and even IRC transcripts (see the introduction for links to some of these). More often than not these discussion are not focused on REST per se, but on anything related to the Web, various specifics concerning current and future standards and so on. And even more often, the discussions unnecessarily repeat previous discussions probably because the authors are unaware that the same discussion exists elsewhere or it's that those are too difficult to search for. For example, the HATEOAS constraint is getting a lot of attention everywhere. So there's lots of unorganized, scattered, duplicated and mutually disconnected fragments on REST and it's unclear where and how to find the ones that answer your questions. No, Googling won't help as often as you'd like, no, a series of unrelated articles piled up together won't either (though Mike Amundsen's tweets often will), and yes, a-big-unorganized-mess is kind of the point of the Web in general, however it's not the optimal methodology for systematizing knowledge in a way required for understanding complex concepts like REST.

Third, these fragments on REST often use mutually different terminology, same terminology with different meaning or terminology which is not explicitly defined. My favorite example is the word "state" which is overloaded with unexplained and overlapping meaning. "Resource state", "session state", "control state", "client state", "server state", "application state", "transaction state", "steady state" and "transient state" are some of the terms used and it is not completely clear what some of those mean, how they are related and which REST elements use, change and transfer them and when. For example, what is client state? Is it something related to the client connector type? Is it something stored on the client component? If so, then why is it not called user agent state? Is client state a union of session and application state? Or is application state a synonym for session state? Which states does a steady state relate to - client, session or application? Which entities may change session state and when? Yes, some of these concepts are completely clear, but some are not and different people use them with different meaning. Still, the most entertaining situations are the ones in which people write only "state" and then you have to figure out which state they are referring to. Yeah, good luck. And there are other both simple and complex examples, from stating that REST methods are to be executed over representations instead of resources and discussing if ATOM is RESTful or not to explaining just what is an application in terms of REST. Lastly, similar to Roy's dissertation, these discussion rarely use diagrams or formal models to explain anything.

These problems confuse people trying to learn REST on a basic level and make it hard to discuss REST on deeper levels.

What we should do about it

Now, I'm not implying that these problems will cause the apocalypse if we don't solve them, but I'd sure like it if they go away (and lots of other people would also). So here are my suggestions:

First, a mess is still better than nothing. Everyone should at least continue with thinking, writing and talking about REST and create more fragments. Chances are that those fragments will help a fair amount of people interested in REST. I definitely learned a lot from reading excellent blogs, tweets, papers and mailing list posts (Ryan's explanation of REST to his wife is especially entertaining, besides educational).

Second, I think this problem can and should be solved collaboratively and openly, not by single person (not even Roy Fielding) or as just another academic paper. If there's going to be agreement over terminology, it's meaning, models, and other ways of making understanding REST easier, this agreement must be backed by people who have relevant REST experience and must be open to comments from everyone else. Furthermore, untangling the mess will require a lot of work. So, in my opinion, the REST community (whatever that is) should:

Agree that there is a problem worth fixing - do we think that we can create a better, clearer and more systematized way of understanding and discussing about REST?
Express interest in fixing it - is this something people want to contribute their time to?
Agree on how to fix it - what should be our output (a RESTopedia, a document, video tutorials) and how would we all contribute to and moderate the process?
Do it - spend time discussing and developing the output.
Eat our dogfood - use whatever we produce. If we don't use the terminology and models we agree upon, the the mess has only gotten bigger.

There are more than enough smart and motivated people with different backgrounds and experience with REST and RESTful HTTP to make this happen and it would have a big impact if done (even more if done right). I myself am not sure what the best output would be and how to achieve it, but would like it to be a hypermedia document available freely on the WWW, contain a systematized intersection of terminology, images, models and rationale that everyone agrees upon and focus on REST, using HTTP/URI/HTML only for examples (not the other way around).

Formal models for REST - a FSM model of a simplified RESTful user-agent

First off, developing useful formal models for understanding architectural styles isn't easy since architectural styles are named sets of constraints commonly defined using natural language, rarely using formalisms. Also, models of REST concepts should include as many other REST concepts as possible - e.g. a model of HATEOAS should somehow clear up how application state, representations, methods, resources, steady states, transient states and other concepts all play their role in HATEOAS. It's not easy to model this in a clear and simple way. Nevertheless, these kinds of models are especially important since they connect rather than disperse concepts.

Mike Amundsen's and Jan Algermissen's poking at REST steady states was very thought-provoking for me - excerpts pointing at something often ignored when discussing REST, but could be formally defined. This led me to try to model REST user-agents using some kind of a state machine. I soon found out there are lots of interesting papers on modeling hypermedia applications in general and even more blogs on modeling REST user-agents using some kind of state machine formalism (FSMs, statecharts, Petri nets or something else). Here are some of my thoughts after reading these papers and blogs:

Most often, models are based on the "each page is a state and each link is a state transition" analogy. This is confusing and wrong for two reasons. First, it's confusing since resource identifiers (e.g. URIs) are used to somehow address states of the application (e.g. an URI addresses a page which is mapped to a state) and state transitions (e.g. an URI is a link to a state which is mapped to a transition). In REST, resource identifiers are neither the single thing determining the state (e.g. two clients can perform the same GET request on a resource and get different representations which determine state) nor the single thing determining the transition between states (e.g. one client can perform a GET on a resource, the other could perform a PUT). Roy's thesis defines what constitutes application state and thus when the state changes: "An application's state is therefore defined by its pending requests, the topology of connected components, the active requests on those connectors, the data flow of representations in response to those requests, and the processing of those representations as they are received by the user agent." Second, models often ignore some REST concepts like steady and transient states and are therefore useful only for understanding REST up to a certain level.
State machines models are extremely simple to understand and a powerful tool for modeling. For example, state machines can also be used for model checking using temporal logic (e.g. check that from any state there is a link to the home page).

That's why I really like Stu Charlton's recent post on RESTful user-agents which identifies different types of state machines involved in user-agent operation. Stu nailed most of the things I wanted to write about, but nevertheless - below is my first attempt at a simplified finite state machine (FSM) model of RESTful user-agents. I won't go into explaining FSMs in detail, so just check the Wikipedia article if you don't know what FSMs are.

I'll concentrate on recognizer-type finite state machines - specifically, nondeterministic FSMs (NFAs). NFAs are mathematically defined as a quintuple (Σ,S,s₀,δ,F), where

$Σ$ is the input alphabet (a finite, non-empty set of symbols).
$S$ is a finite, non-empty set of states.
$s 0$ is an initial state, an element of $S$ .
$δ$ is the state-transition function: δ : S x Σ → P(S) where P(S) the power set S.
$F$ is the set of final states, a subset of $S$ .

The operation of an NFA automaton, as state machines are sometimes called, is described as follows. The NFA starts from the initial state and then sequentially reads one input symbol at a time each time applying the state-transition function to transfer itself to the next state. After all input symbols have been processed in this way, the NFA stops and outputs "true" if the last state is in the set of final states, or "false" otherwise. Notice that it is not defined how input symbols are generated, just that there is a sequence of them being fed to the automaton. Also notice that NFAs can nondeterministically transfer to a set of states at some point. This nondeterminism is important but also confusing - how can the automaton be at more than a single state? Well, although the formal definition indicates that the automaton is in a set of states, the practical and useful meaning is that the automaton may be at any single state from that set - we don't know which one in advance, it can be any single one. In other words, a single state from the set will be chosen in some way e.g. using probabilities.

This iterative process of automaton operation can be modeled as a system:

CurrentState = s₀
while there are more input symbols to process:
  InputSymbol = GetNextInput()
  CurrentState =
    TransitionFunction(InputSymbol, CurrentState)

Here, CurrentState is a component that stores the current state of the automaton, InputSymbol is a component that stores the current input symbol (element from the input alphabet), GetNextInput is a component that provides the next input symbol and writes it to the InputSymbol component and TransitionFunction is a component that computes the state transition function and writes the new state to the CurrentState component. Notice that the GetNextInput component doesn't take any input in this description. This is not entirely true, of course - the operation of the component that generates input symbols is out of scope of formal automaton definition, the component may generate input symbols based on whatever it wants. So, I'll expand the system to include the CurrentState as an input to the GetNextInput component (since there is nothing else in the model to include):

CurrentState = s₀
while there are more input symbols to process:
  InputSymbol = GetNextInput(CurrentState)
  CurrentState =
    TransitionFunction(InputSymbol, CurrentState)

Now the interesting part - mapping properties of RESTful user-agents to this NFA model. As I wrote before, this is a simplified first-draft-quality model, not everything that should be included is included and thus the model may change when more details do get included. First I'll try to explain the mapping in a general way, not on a specific RESTful system, and after that I'll give a concrete example:

The input alphabet (set of input symbols generated by the GetNextInput component and stored in the InputSymbol component) are REST requests. Since a REST requests is defined by a resource identifier, method to be performed on the resource, metadata and a possible representation, the input alphabet is the set of all possible (syntactically) well-formed REST requests. This is why the resource identifier is not the only thing representing a transition - a single transition from a specific state is defined by all elements of a REST request, only one of which is the resource identifier. Therefore, there may be multiple transitions from a specific state and a specific destination resource identifier.
The state transition function, implemented in the TransitionFunction component, is the cumulative processing of REST requests. This process is performed by user-agents, intermediaries and origin servers, and the result is a REST response containing metadata and a representation of the resource identified in the related REST request. Here's the most important part - since the user-agent doesn't know what the response to a request will be and since the origin server may return different REST responses for the same REST request - this processing must be modeled as nondeterministic. In other words, the result of the transition function is a set of all possible REST responses which may be returned for a given REST request. As I explained earlier when demystifying nondeterminism, the origin server will in fact return a single response but since we don't know which one - this is modeled as a set of possible responses. Also, the metadata (media type) of the response defines if the state is steady or not by specifying which of the resources linked to from the response representation should also be requested. If the state is not steady, more requests should be sent for those resources. If the input symbol is a REST request for a resource for which there is no link in the current state then the transition function returns an empty set. The same happens if the system is not in a steady state and the input symbol is a REST request not in the set of pending requests.
The current state, stored in the CurrentState component, is a set of REST responses and pending REST requests. The current state is considered a steady state if there are no pending requests, otherwise it's considered a transient state. The initial state of the user-agent is steady and may contain a predefined representation with links to bootstrap the operation of the user-agent.
The only thing left is to define how the user-agent chooses transitions - the next REST request for the current state. This is the role of the GetNextInput component which implements both application- and hypermedia-level logic. The application-level logic is in charge of generating input symbols in case the current state is steady, or in other words, it chooses then next step for achieving the overall application goal. The application-level logic may be a software program or a program that delegates this responsibility to a human user. The hypermedia-level logic is in charge of generating input symbols in case the current state is transient, or in other words, it chooses which of the pending REST requests will be processed next. To satisfy the HATEOAS constraint, both the application- and hypermedia- level logic generate REST requests with resource identifiers linked to from the current state.

Or mathematically (without all the detailed explanations):

$Σ$ = { R | R is a valid REST request consisting of a resource identifier, method, metadata and representation }
$S$ = { { G } | G is a valid REST response consisting of metadata and a representation or a pending REST request }
$s 0$ = initial representation containing links to bootstrap the operation of the user agent
δ : S x Σ → P(S) where P(S) the power set of S.
$F$ = { Z | Z ∈ S and Z is a steady state }

The operation of the automaton can be described as follows. The automaton starts from the initial steady state containing links for bootstrapping the application. Each time the automaton is in a steady state, the application level logic generates the next input symbol (request) based on the links in the current state representations and the overall application goal. The request and the current state are used by the processing infrastructure (user-agent, intermediaries, origin server) to generate a response. Notice here that the current state (representation already on the client) need not be sent to the server, it may be used on the user-agent for processing (e.g. to determine if the request is for a resource linked to from the current state or not) while only the input symbol (the request) is sent over the network. The result of the processing is the new state of the user-agent. The new state may either be steady or transient, based on the representation received in the response and it's metadata (media type). If the state is transient, the hypermedia-level logic automatically chooses requests for fetching other resources in order to bring the user-agent into a steady state so that application-level logic can take over.

Now an example - applying this model to a specific RESTful system, which will of course be a simple web application shown above. The purpose of the web application is to simulate coin tossing. It has two web pages, Main.html and Cointoss.html. The Main.html page contains only a single link to the Cointoss.html page and nothing else. The Cointoss.html page has a link back to the Main.html page and a single image. Since the web application simulates coin tossing, which is nondeterministic, the image which gets included into the Cointoss.html is chosen randomly by the server and is either the heads.png image or the tails.png image.

The input symbols of the resulting automaton are requests for fetching application resources - the html pages and the images. The states of the resulting automaton are both steady states representing a complete page load or transient states representing partial page loads. Therefore, the Main.html page is represented by only a steady state since it only contains a link to Cointoss.html, while the Cointoss.html page is represented by both a transient state and a steady state since it contains an image which must be fetched after the initial page. Let's assume that the initial state contains a single link to the Main.html page. The transition function of the resulting automaton defines which responses the processing infrastructure returns for each state and requests. Therefore, if the user-agent requests a page or image, the server returns the requested page or image. However, if the user-agent is requesting the Cointoss.html page - it isn't known in advance what the server will return (heads or tails), so we need a nondeterministic transition into two possible states. Lastly, steady states define the set of acceptable states.

And here's the formal automaton definition for the explanation above:

$Σ$ = { R1, R2, R3, R4 }, where R1 is a request for fetching Main.html, R2 is a request for fetching Cointoss.html, and R3 and R4 are requests for fetching the heads.png and tails.png images. For simplicity, let's say these are the only input symbols the user-agent may generate, but in general - it may generate other syntactically valid requests.
$S$ = { S1, S2, S3, S4, S5, S6}, where S1 is the initial state containing a link to Main.html, S2 is the state after receiving the Main.html page, S3 is the state after receiving the Cointoss.html page which has a link to the heads.png image which hasn't been fetched yet, S4 is the state after receiving responses to both the Cointoss.html page and heads.png, S5 is the state after receiving the Cointoss.html page which has a link to the tails.png image which hasn't been fetched yet and S6 is the state after receiving responses to both the Cointoss.html page and tails.png.
$s 0$ = S1, since S1 is the initial state.
The transition function is defined for each pair of automaton state and input symbol: δ(S1, R1) = S2, δ(S2, R2) = {S3, S5}, δ(S3, R3) = S4, δ(S4, R1) = S2, δ(S5, R4) = S6, δ(S6, R1) = S2. For other pairs of state and input symbol the function returns an empty set { } since there are no other links for those pages (states) to resources identified in those requests (input symbols).
$F$ = { S1, S2, S4, S6 }, since those states are steady.

And here's the equivalent state diagram visualization of the automaton:

For such a model, we could generate a random sequence of input symbols and check if the automaton will end up in a steady state or not. You can do this for homework :).

Conclusion

(EDIT: inserted a few sentences to make this a real conclusion) That's about it. All-in-all, I think REST is an important concept for the WWW and that more work should be done to explain it fully and concisely. After that, we can simplify the models to reflect the most important properties, but simplifying before understanding leads to ignorance and misunderstanding in discussions.

If you have ideas or comments about systematizing knowledge, terminology and models for REST or developing formal models for REST - let me know. @izuzak

Real-time feed processing and filtering

2010-01-11T00:00:00-08:00

This is another lengthy post so I'm writing a brief overview as an introduction.

Introduction

This post is about web syndication feeds (RSS and ATOM), technologies for real-time delivery of feeds (PubSubHubBub, RSSCloud) and two opportunities I believe could help make these technologies better and widen their adoption: real-time feed processing/filtering and end-user selection of processing/filtering services. First, I'll write about my experience with polling-based feed processing services from developing Feed-buster, a feed-enhancing service for use in FriendFeed. Next, I'll give a short overview of PubSubHubBub (PSHB) and several ways of integrating feed filtering and processing functionalities into the PSHB ecosystem without changing the PSHB protocol. Lastly, I'll try to argument why there should be a better solution and outline an idea for an extension to the PSHB protocol.

Before I continue, I have to acknowledge that I've discussed some of these ideas with Julien Genestoux (the guy behind Superfeedr) and Brett Slatkin (one of PSHB developers) several months ago. Julien and I were even starting to draft a proposal for a PSHB extension for supporting filters, however, due to other stuff in our lives, we didn't really get anywhere.

Polling-based feed processing and filtering - experience from FriendFeed and Feed-Buster

FriendFeed is a social-networking web application that aggregates updates from social media sites and blogs that expose an RSS or Atom feed of their content. You give FriendFeed the URI of an RSS/Atom feed and it fetches and displays short snippets of updates as they appear in the feed. In most cases this means that only the title and a short snippet of the description of the new item are shown. For feeds that have special XML tags that list media resources in the update, the images/videos/audio content from the new feed item is also displayed. The problem is that most RSS/Atom feeds don't have these media XML tags since webapps that create the feeds (e.g. blogging engines) don't bother defining them. This gave a not-very-user-friendly look to most FriendFeed user pages which were just a long list of update titles and possibly snippets.

So I built Feed-buster, an AppEngine service that inserts these MediaRSS tags into feeds that don't have them. Since the only thing FriendFeed accepts from users are URIs pointing to feeds - the service must be callable by requesting a single URI with a HTTP GET request and must return an RSS/Atom feed in a HTTP response. Here's how Feed-buster works:

An URI to an RSS/Atom feed is supplied as a query parameter of a HTTP GET request to the service. E.g. http://feed-buster.appspot.com/mediaInjection?inputFeedUrl=http://myfeed.com/atom.
The service fetches the target feed, finds all the images, video and audio URIs and inserts the special XML tags for each one back into the feed.
The service returns the modified feed as the content of a HTTP response.

To use the service, the user would obtain an URI of an RSS/Atom feed, create the corresponding Feed-buster URI by appending the feed URI to the Feed-buster service URI and pass it to FriendFeed (1). Suppose that the feed is not real-time enabled (i.e. a PSHB or RSSCloud feed). In this case, every 45 minutes FF will poll for updates - send a HTTP GET request to the service (2), the service will fetch the original feed (3), process the feed (4) and return a HTTP response with the processed feed (5).

Feed-buster is a feed processing service: it takes a feed URI as input and returns the feed it points to as output, potentially modifying any subset of feed items (but not removing any). Feed filtering services are similar: they take a feed URI as input and return the feed it points to as output, potentially removing any subset of feed items (but not modifying any). The difference is subtle and both terms could be unified under "feed processing". Furthermore, this model of feed processing/filtering was nothing new at the time I developed Feed-buster, there were numerous services that filtered/processed feeds based on user preferences. The best example is probably Yahoo! Pipes which enables anyone to create a filtering/processing pipe a publish it as a service just like Feed-buster.

Lessons learned:

Feed-buster is very simple to use and this helped to make it popular (appending an URI to another URI is something everyone can do).
Feed-buster's model of use is based on separation of filtering/processing services as individual entities in the ecosystem which enables that these services be developed independently of publishers and subscribers. Applications that enable other applications make ecosystems grow.
Polling is bad (1). Because AppEngine applications have a fixed free daily quota for consumed resources, when the number of feeds the service processed increased - the daily quota was exhausted before the end of the day because FF polls the service for each feed every 45 minutes. See the AppEngine resource graph below for a very clear visual explanation :). Of course, I implemented several caching mechanisms in the service and it helped, but not enough.
Polling is bad (2). Real-time feed delivery arrived in the form of PubSubHubBub and RSSCloud, everyone loved it and wanted everything to be real-time. But Feed-buster feeds weren't real-time. Even if the original feed supported PSHB, the output feed didn't (since it Feed-buster wasn't a PSHB publisher or hub). Feed-buster users were not happy.

PubSubHubBub - real-time feed delivery

PubSubHubBub is a server-to-server publish-subscribe protocol for federated delivery of feed updates. Without going into a lot of detail, the basic architecture consists of publishers, subscribers and hubs which interact in the following way:

A feed publisher declares its hub in its Atom or RSS XML file and a subscriber initially fetches the feed to discover the hub.
The subscriber subscribes to the discovered hub declaring the feed URI it wishes to subscribe to and an URI at which it wants to receive updates.
When the publisher next updates the feed, it pings the hub saying that there's an update and the hub fetches the published feed.
The hub multicasts the new/changed content out to all registered subscriber.

And this gives us real-time feed delivery. But what about real-time filtering and processing? How would a real-time-enabled version of Feed-buster be built? Of course, there are a lot more reasons for doing this, both from business and engineering perspectives.

Real-time feed processing and filtering (take 1)

So, the idea is to extend the PSHB ecosystem with filtering/processing services in some way. First, here are a few ideas for doing this without extending the PSHB protocol:

Integrating filtering/processing services into publishers. This means that the publisher would expose not only the original feed, but also a feed for each filter/processing defined by the subscriber. For example, Twitter has a single feed of all updates for a specific user, a feed for updates from that user containing the string "hello", updates not containing "world", updates written in English, and so on.
Integrating filtering/processing services into subscribers. This means that the subscriber application would subscribe to feeds and filter/process them on it's own. It's not exactly the same thing but, for example, FriendFeed subscribes to feeds you want it to aggregate and later you can filter the content from within the FriendFeed application -- this is called search, as in Twitter, but basically means filter.
Integrating filtering/processing services into hubs. This means that a hub would not only deliver feed updates, but process and filter them also. This would cause a large number of hubs to be developed, each implementing some filter or processing functionality. If you want to filter a feed - subscribe to a hub that implements that filter. If you want a new filter - build a new hub. However, if an end-user tells a subscriber application to subscribe to a feed with a specific URI, the subscriber will subscribe to a hub specified in the feed, not to the filtering/processing hub (since that hub wasn't specified in any way). A possible way for enabling end-users to select which hub they want to use is to generate dummy feeds on filtering/processing hubs (e.g. http://www.hubforfiltering.com/http://www.myfeed.com/atom). The contents of this dummy feed is the same as the contents of the original feed but with a simple difference - the hub elements from the original feed are removed and a hub element for new hub is inserted. This way, when the end-user passes the dummy feed URI to the subscriber application, the application will subscribe to the filtering/processing hub, not the original one. The filtering/processing hub then subscribes to the original hub to receive updates, filters/processes received updates and notifies the subscriber.
Masking the filtering/processing services as the subscriber which then notifies the real subscriber. I'm not sure this is possible (too lazy to go through the PSHB spec), but in theory, here's how it would work. The subscriber would obtain an URI pointing to the filtering/processing service for which the service would know what the real subscriber notification callback is (e.g. http://www.filter.com?subscriber=http://www.realsubscriber.com/callback). The subscriber would subscribe to the hub using the obtained URI. When the publisher publishes an update, the hub would pass the update to the filtering/processing service thinking it was the subscriber. The filtering/processing service would do it's thing and pass the update to the real subscriber. It's unclear though how this would be done by end-users since they'd have to know the subscriber application's callback URI.

All of these approaches have their pros and cons. Some are not easily or at all end-user definable, some may introduce unnecessary duplication of notifications, some introduce delivery of unwanted notifications, some do not support reuse of existing infrastructure and cause unnecessary duplication of systems, and most do not promote the development of filtering/processing services as a separate part of the ecosystem.

Real-time feed processing and filtering (take 2)

What I believe is needed is an architecture and protocol which supports three things. First - filtering/processing services as a separate part of the ecosystem. Since publishers and subscribers can't possibly anticipate and implement every possible filter and processing service, this will enable a stream of development activity resulting in services with different functionality, performance and business models. Again, applications enabling other applications help software ecosystems grow.

Second - a scalable processing model where hubs invoke filtering/processing services. Regarding architectural decisions on where filtering/processing should be applied, there's a paper (citation, pdf) from 2000. on "Achieving Scalability and Expressiveness in an Internet-scale Event Notification Service" which states two principles:

Downstream replication: A notification should be routed in one copy as far as possible and should be replicated only downstream, that is, as close as possible to the parties that are interested in it. Upstream evaluation: Filters are applied and patterns are assembled upstream, that is, as close as possible to the sources of (patterns of) notifications.

In short, these principles state that filtering services should be applied as close to the publisher as possible so notifications that nobody wants don't waste network resources. However, processing services should be applied as close to the subscriber so that the original update may be transported through the network as a single notification for as long as possible. In cases where there's a single hub between the publisher and subscriber, the same hub would invoke both processing and filtering services. However, in more complex hub network topologies different hubs would invoke different types of services based on their topological proximity to the subscriber/publisher.

Third and last - end-user selection of these services. When 3rd party filtering/processing services exist - how will they be chosen for a particular subscription? In my opinion, this must be done by end-users and thus there must be an easy way of doing it. The first thing that comes to mind would be to extend the subscription user-interface (UI) exposed to users of subscriber applications. E.g. instead of just showing a textbox for defining the feed the users wants to follow, there could be an "Advanced" menu with textboxes for defining filtering and processing services.

No matter how cool I think this would be as a core part of the PSHB protocol, it's probably a better idea doing it as an extension (as a first step?). So here's what the extension would define:

The subscription part of the protocol additionally enables subscribers to define two lists of URIs when subscribing to a hub, one declaring the list of filtering services and the other the list of processing services that should be applied.
Hubs are responsible not only for delivery of notifications but also for calling filtering/processing services on those notifications and receiving their responses. An inter-hub subscription negotiation/propagation mechanism that enables the optimal placement of subscription-defined filters and processing services in the delivery flow. In case of a single hub, nothing needs to be done since the hub invokes both types of services. However, in case of two hubs, the hub which first received the subscription would locally subscribe to the second hub stating only the filtering services while processing services would not be sent upstream. Thus, the hub closer to the subscriber would be responsible for invoking processing services, and the hub closer to the publisher would be responsible for invoking filtering services. This could, I believe, be generalized and formalized for any kind of hub network topology.
When an update is published, hubs responsible for invoking filtering or processing services must do so before forwarding the notifications downstream. There are several performance-related issues which need to be addressed here. First, the invocation of filtering/processing services should be asynchronous, webhook-based (the hub calls a service passing an URI to which the result of processing should be POSTed). Second, invocation of services on a specific hub for a specific notification should be done in parallel, except if multiple processing services were defined for a single subscription in which case those processing services must be invoked serially (filters can always be invoked in parallel). Third, there's a not-so-clear trade-off between a) waiting for all invoked filters to respond and sending a single notification carrying all metadata regarding which filters it satisfied and b) sending multiple notifications containing the same content but different metadata regarding which filters were satisfied (e.g. as each filter responds). Since all filters will not respond at the same time (some will respond sooner, some later and some possibly never) - waiting for all filters invoked in parallel before continuing seems contradictory to the constraint of real-time delivery. On the other hand, creating multiple notifications with the same data will introduce an overhead in the network. Overall, I believe that the overhead will not be substantial since there will not be many duplicates -- this can even be regulated as a hub-level parameter e.g. one notification for services returning in under 2 seconds, one for under 10 second, and one for everything else, and the benefit is basically a necessity.

Conclusion

Of course I didn't consider all aspects of the problem and there are a lot more points of view to be considered, from is it legal to change content in a feed and in what way to how do we make this extension secure and fault tolerant? Nevertheless, I believe that the need and expertise are here and that it's an opportunity to make the real time web better. What are your ideas for extending real-time content delivery technologies? @izuzak

RPC for Web Workers and distributed computing within the browser

2009-12-21T00:00:00-08:00

In a previous post I wrote about pmrpc - a project of mine for developing a JavaScript library for cross-domain inter-window RPC-style communication in HTML5 browsers. RPC communication is based on two technologies: HTML5 cross-domain messaging as the transport mechanism and JSON-RPC as the RPC protocol and message format, while the library supports other advanced features (access control, asynchronous calls, retries...). At the end of the post I outlined a few things Marko and I wanted to implement next in pmrpc, one of which was support for Web Workers. From the post:

Web Workers are another specification being developed in parallel with HTML5. The specification “defines an API for running scripts in the background independently of any user interface scripts”. What’s interesting is that the postMessage API is used to communicate with web workers and therefor pmrpc could be extended to include support for web workers.

Well, the API used to send messages from a web page to a worker and vice versa is not exactly the same as the cross-document messaging API, but is very similar. So, the idea for supporting Web Workers was leave the pmrpc API for registering, calling and unregistering procedure unchanged, and just extend it to also use the Web Worker API for message transport (in addition to the cross-doc messaging). And about two weeks ago we released a new version of pmrpc with Web Worker support. Here's an example of using pmrpc with Web Workers:

 1 // [worker object A - testingWorker.js]
 2 
 3 // load pmrpc library
 4 importScripts('pmrpc.js');
 5 
 6 // expose a procedure
 7 pmrpc.register( {
 8 publicProcedureName : "HelloPMRPC",
 9 procedure : function(printParam) {
10               alert(printParam); } } );

 1 // [window object B]
 2 
 3 // load pmrpc library
 4 <script type="text/javascript" src="pmrpc.js"></script>
 5 
 6 // create worker
 7 <script type="text/javascript">
 8   var testingWorker =
 9     new Worker("testingWorker.js");
10 
11   // calls the exposed procedure
12   pmrpc.call( {
13   destination : testingWorker,
14   publicProcedureName : "HelloPMRPC",
15   params : ["Hello World!"] } );
16 </script>

A few notes on worker support. Pmrpc currently supports only normal ("dedicated") Web Workers. Shared Workers (workers shared between multiple web pages) and nested workers (workers within workers) are currently not supported since, as far as I know, Shared and nested Workers are not implemented yet in any browser. Also, since Workers must be loaded from the same origin as the page that created them - the access control feature in pmrpc make no sense and are therefore not supported for Web Workers.

Next on our list is enabling discovery of registered procedures scattered over multiple browser workers, iframes and windows. Currently, pmrpc requires that you tell it where the procedure you are calling is, but sometimes you don't know where it is, or don't care - e.g. you just want to call the "DrawMap" procedure loaded from domain "http://www.good.com" if it exists. Discovery should enable discovery of methods based just on their names. The problems in implementing this feature (knowing which windows, iframes and workers are currently running and syncing the list of registered procedures with each of them) made me think about where systems running in browsers are heading - towards a form of distributed & parallel computing, only within a single browser. Distributed and concurrent execution, discovery, access control, synchronization, remote invocation, fault tolerance and naming are all concepts well know in distributed computing, and will possibly be re-invented in the context of web browsers to support this new shift. I'm not sure how far pmrpc will go in this direction, but I hope it will be a good example for other developers and research projects.

Do you think that web browsers need more powerful mechanisms supporting this new form of distributed computing? @izuzak

Extending and customizing Google Code project hosting

2009-12-13T00:00:00-08:00

I've used Google Code project hosting for almost every project I've started in the last few years. It's really great and I love both the Web UI and all the functionality-related features (which Google is continuously extending). However, one aspect in which I've found Google Code project hosting strangely lacking is customizability and extendability in adding rich content and even external applications to a project site. Wouldn't it be great if we could add a special "Testing" and "Group discussion" tab or an easy way to embed presentations and JavaScript code into wiki pages? This would boost the usability of project sites and make the visitor experience so much better.

So, I've been thinking about this idea and playing with hacks and workarounds for extending Google code sites for my projects, and what I've discovered is that I'm not alone - several other projects have been doing this also. So in this post I'll try to cover two approaches to extending and customizing Google Code project hosting: embedding 3rd party content using Google Gadgets and viewing non-source files in the repository using the SVN mime-types property.

The first idea for extending project sites is to embed 3-rd party content into wiki pages (Google Docs presentations, JavaScript programs and such). Although it is possible to use a large subset of HTML in project wiki pages, <iframe> and <script> tags are not available. However, embedding Google Gadgets is supported. Google Gadgets are " simple HTML and JavaScript applications that can be embedded in webpages and other apps" (think clock, calendar, twitter and similar widgets) and are, in essence, an HTML <iframe> embedded in the parent page. Although there is a public directory of existing Gadgets which you could add to your wiki pages, what's more interesting is that you can build a Gadget yourself and that it's really easy to do. So, finally, the first idea for extending Google Code project hosting is to build your own Google Gadgets which wrap the content you want to embed in your wiki page. Here's what you need to do:

Create a file containing a blank, content-less XML Gadget specification -- you can just copy/paste the code from the Google Gadgets tutorial.
Define the content of the Gadget. Depending on what you want to embed in your wiki page, there are two ways to go here:
1. If you want to embed an existing web page or rich content available on the Web, the content of the Gadget will be a single <iframe> element with the src attribute pointing to the URL of that web-page.
2. If you want to embed content which does not exist on the web, you will have to write the HTML + JavaScript code that defines that content yourself and inline it in the Gadget. Be sure to check out the Google Gadgets developer guide.
Save and upload the Gadget XML specification file into your project repository.
Edit one of your wiki pages and create a Google Gadget element that references the URL of the uploaded specification. Note that you must reference the raw-file URL of the XML spec which you can get by viewing the uploaded file in your project's Source tab and then clicking "View raw file" under File info.


1. PubSubHubBub	2. Feed-buster

3. Pmrpc testing	4. UrlEcho

And here are some examples that show the usefulness of this approach (images above, in order of appearance):

Embedding Google Docs presentations that visually explain details of your project (example project wiki page and XML spec). To get a Google Docs URL which you can use for embedding, go the presentation's page in Google Docs, click "Share - Publish/Embed", and copy the URL under "Your document is publicly viewable at:".
Embedding a FriendFeed discussion groups for in-place discussions (example project wiki page and XML spec -- if you see a blank page, view the source of the page). To get a FriendFeed group URL which you can use for embedding, use the official FriendFeed instructions (basically, it's http://friendfeed.com/GROUP_NAME_HERE/embed e.g. http://friendfeed.com/feed-buster/embed).
Embedding QUnit testing pages for following project stability and cross-browser testing (example project wiki page and XML spec). This one is a bit complex -- first you'll need to create and upload a HTML page containing the QUnit test page (setting the mime-type property to text/html using instructions below) into your repository, and then link to the raw page URL in the <iframe> inside your Gadget.
Embedding dynamic content and forms (example project wiki page and XML spec). There is no outside content in the Gadget spec here, everything is in the Gadget. Note that the XML spec in the example also contains CSS styles so that the Gadget content has the same look-and -feel as the rest of the wiki page so visitors won't event know that a Gadget is there.

The second idea for extending project sites involves using the source repository for hosting non-source files which will be viewed by project site visitors. This is useful if you want the freedom of creating custom web pages (not embed them into project wikis) and if you want to keep all those files in the project repository (and not on some other server). The idea here is to use the raw-file URL for files in the repository - visiting the raw-file view URL returns only the content of the file without the UI stuff of Google Code. The problem is that, by default, any file uploaded to the repository is attributed to be of type text, so when you view the raw-file URL in the browser you get the content as text, which is useless if you want to see a HTML page. The good thing is that using custom SVN per-file properties it is possible to set the type of the file when it is uploaded in the repository. So, finally, the second idea for extending Google Code project hosting is to upload your content files into the repository and explicitly set their mime-types to the correct value. Here's what you need to do :

Create your content and save it into a file on your hard drive in the project repository folder.
Right-click the file and under TortoiseSVN -> Properties click "New". In the "Property name" drop-down list choose "svn:mime-type" and ender the correct mime type for that file. You can find a list of all mime types here. E.g. for HTML pages it's "text/html" (this is for TortoiseSVN, command-line version is here).
Upload the file to the repository.
Obtain the URL to the raw-file by viewing the uploaded file in your project's Source tab and then clicking "View raw file" under File info.
Create a link to the obtained URL from one of your wiki pages.


1. PubSubHubBub specification	2. Closure library API docs

And here are two examples that show the usefulness of this approach (images above, in order of appearance):

Creating a protocol specification with custom styling (example project page and source file)
Creating advanced API docs (example project page and source file -- note: you will not be able to view the source file, access is being blocked somehow)

And that's it. Go fancy-up your project sites and let me know if you know any other cool ways of extenting Google Code project hosting. @izuzak

Inter-window browser communication and how to make it better

2009-10-10T00:00:00-07:00

As promised in my post on named arguments in JavaScript, this post will be about pmrpc - a project I've been working on with Marko for the last few months.

But first, a bit of background. From mid 2007. when I was interning at Google up until today and my dissertation-related research, an issue that's consistently been popping up in anything I worked on is inter-window communication. Inter-window communication (IWC) is the process of transporting data from one window object within a browser to another window object inside the same browser. What's relevant here is that window objects are both full-size windows and iframes - window objects that can be embedded within other window objects (think of an iGoogle page and all the gadgets in it).

The reason why IWC is an issue is the security model of communication within browsers. Same Origin Policy (SOP), a security policy introduced to reduce security risks, restricts access of processes executing within browsers. In essence, SOP prevents a document or script loaded from one domain from getting or setting properties of a document from another domain. For example, a page loaded from http://www.google.com can not directly access data in an iframe loaded from http://www.yahoo.com.

Although SOP turns Web applications into a set of isolated units, the need for communication between those islands is growing rapidly. With the advancement web technologies, web applications were slowly becoming componentized and many current web applications are based on mashing-up content from different sources - map gadgets, finance widgets, blog commenting toolboxes, games, chat widgets and many, many others. Since widgets, content encapsulated within an iframe, are the most popular method of including cross-domain content on a web page, inter-widget communication is most often implemented as inter-window communication.

A lot of groups and projects are researching models and developing systems that address the problem of communication between widgets and between windows in general. So here's a short overview of this broad space of R&D. Various widget specifications are trying to formally define and standardize what a widget actually is. Browser security in general and various issues concerning intra-browser communication and mashup security are being heavily researched. Low-level IWC mechanisms are being proposed and implemented, sometimes even as browser hacks, and high-level communication models are being researched and developed. Even patents have been filed for methods of inter-window and inter-widget communication.

Among all of these efforts, one is catching on on a wider scale - the HTML5 cross-document messaging specification aka the postMessage API. The postMessage API has been implemented in all popular web browsers and enables secure message-oriented communication between any window objects - scripts in any window may send messages to any other window and receive messages from scripts in other windows. Security is established by two access-control mechanisms: 1) the message sender may specify the domain from which the destination window must be loaded in order for the message to be delivered, and 2) the message receiver may inspect the domain from which a message was sent and drop unwanted messages.

The problem with postMessage is that it's a very low-level and general communication mechanism which requires a lot of boiler-plate code every time it is used and even more if one wants to implement advanced communication or access-control models. Have a look at the example on using postMessage in the HTML5 spec to see what I'm talking about. However, it's exactly this generality and low-levelness that makes postMessage an excellent foundation to build upon! Some of the projects I mentioned above have been working on advanced communication libraries based on postMessage, but none that I know of are open-source and applicable in any context but rather tied to a specific application environment. Furthermore, advanced access control, fault tolerance and many other aspects of inter-window communication have been under-explored. These drawbacks were the main motivators for developing pmrpc.

Pmrpc (postMessage remote procedure call) is a JavaScript library for advanced inter-window cross-domain communication. The library utilizes the postMessage API as an underlying communication mechanism and extends it to a remote procedure call (RPC) model using JSON-RPC. JSON-RPC is a transport-independent protocol that uses the JSON data format for formatting messages. The pmrpc library provides a simple API for exposing and calling procedures from windows or iFrames on different domains.

Here's a simple example for calling a function in another window:

1. First, a procedure is registered for remote calls in the window or iFrame that contains the procedure:

1 // [window object A] exposes a procedure in it's window
2 pmrpc.register( {
3   publicProcedureName : "HelloPMRPC",
4   procedure : function(printParam) { alert(printParam); } } );

2. Second, the procedure is called from a remote window or iFrame by specifying the window object which contains the remote procedure, name of the procedure and parameters:

1 // [window object B] calls the exposed procedure remotely, from its own window
2 pmrpc.call( {
3   destination : window.frames["myIframe"],
4   publicProcedureName : "HelloPMRPC",
5   params : ["Hello World!"] } );

Furthermore, pmrpc also provides several features for advanced communication: callbacks for successful and unsuccessful calls, access control lists on both the client and server side, fault tolerance using user-defined retries and timeouts for each call, registering asynchronous procedures. The features and their usage are explained in detail on the pmrpc API docs page and a short presentation, so I'll just give an example of an advanced pmrpc call:

1. First, a synchronous procedure named "HelloPMRPC" accessible from windows and iframes loaded from any page on "http://www.myFriendlyDomain1.com" or exactly from the "http://www.myFriendlyDomain2.com" page is registered:

 1 // [window object A] exposes a procedure in it's window
 2 var publicProcedureName = "HelloPMRPC";
 3 var procedure = function (alertText) { alert(alertText); return "Hello!"; };
 4 var accessControlList = {
 5   whitelist : ["http://www.myFriendlyDomain1.com*", "http://www.myFriendlyDomain2.com"],
 6   blacklist : []
 7 };
 8 
 9 pmrpc.register( {
10   "publicProcedureName" : publicProcedureName,
11   "procedure" : procedure,
12   "acl" : accessControlList
13 } );

2. Second, the procedure "HelloPMRPC" is called with a single positional parameter "Hello world from pmrpc client!". In case of success, the return value will be alerted while in case of errors the error description will be alerted. The call will be retried up to 15 times, waiting 1500 milliseconds between retries. Windows or iFrames loaded from any domain, except from http://evil.com, may process the call:

 1 // [window object B] calls the exposed procedure remotely, from its own window
 2 var parameters = {
 3   destination : window.frames["targetIframeName"],
 4   publicProcedureName : "HelloPMRPC",
 5   params : ["Hello world from pmrpc client!"],
 6   onError : function(statusObj) {alert(statusObj.description);},
 7   retries : 15,
 8   timeout : 1500,
 9   destinationDomain :  {  whitelist : ["*"],  blacklist : ["http://evil.com*"] }
10 };
11 
12 pmrpc.call(parameters);

Pmrpc is an open-source project developed by Marko Ivankovic and myself and is available under the Apache v2.0 license. The library works in all web browsers that implement the postMessage API (Firefox 3, Google Chrome, Internet Explorer 8). For more information please visit the project homepage on Google Code and the API docs page for a full description of the API, feature list and usage examples.

I'll end this post with a few ideas on making pmrpc even better:

discovery - currently, in order to call a remote procedure one needs to have a reference to a window that exposes that procedure. Sometimes obtaining a reference to a window is not very practical and easy (or possible) and a way to find windows by various criteria is needed. For example, discover all windows with a specific name or location or discover windows that expose a procedure with a specific name.
web workers - web workers are another specification being developed in parallel with HTML5. The specification "defines an API for running scripts in the background independently of any user interface scripts". What's interesting is that the postMessage API is used to communicate with web workers and therefor pmrpc could be extended to include support for web workers.
publish-subscribe mechanism - there are lots of communication models other than remote procedure call. One of the most popular models is publish-subscribe in which publishers publish events on channels and subscribers subscribe for events on these channels. Messages are delivered from publishers to subscribers without having a direct reference to each other.

I'd appreciate any feedback on pmrpc and our ideas for future work. What are your ideas for the future of inter-window communication? @izuzak

A better future for the FriendFeed tools and services ecosystem

2009-09-02T00:00:00-07:00

FriendFeed is the only social-networking application I get and like using, and one of the reasons why is that it's very community-oriented. The groups and friends approach of segmentation of attention easily directs both topic-oriented discussions and friends-oriented discussions towards the right people.

One other nice thing about FriendFeed is that, although not as popular as Twitter or Facebook, it has a healthy ecosystem of developers extending and building upon the platform. Lots of great user scripts, bookmarklets, feed-processing services, sms services and web apps have been developed and recently even more so with the release of the new API.

However, what I don't like about the ecosystem is the separation of developers which develop tools and their potential users. Developers mostly use the official FriendFeed API Google group and FriendFeed API group for discussing programming issues and the API, while non-programmers mostly join the FriendFeed Google group and the FriendFeed Feedback and FriendFeed for beginners FF groups. This definitely isn't the best solution for the future of the ecosystem. There is no central hub for the exchange of ideas between users and developers - users have a hard time finding existing tools and suggesting new ones while developers are have a hard time finding out what users need.

Enter FF tools and services - a new FriendFeed group for both developers and users to discuss existing and future FF-related tools and services. The goals I hope the group will help catalyze are:

establish a central place where both developers and users of FriendFeed tools can go to sync up and discuss ideas,
compile a single list of all FriendFeed tools and services - if you know or use a FF-related web app, bookmarklet or user script - just add it to this list and other users can use it to find cool apps,
compile a single list of suggestions for future FriendFeed tools and services - if you have an idea for a cool FF web app or service which doesn't yet exist - just add it to this list and a developer with some free time might build it.

And that's it - a place where the FriendFeed tools and services ecosystem can grow. @izuzak

How to supercharge your free AppEngine quota?

2009-08-27T00:00:00-07:00

EDIT (January 25 2012): This post was originally written in 2009. when AppEngine was still a "beta" product. In the meantime, AppEngine graduated and now has a shiny new billing policy. Long story short, Google changed how paid apps work: "Paid apps cost a minimum of $2.10/week". In result, the free lunch is over and all information in this blog post is not valid anymore. Still, I'm keeping the post around for historic reasons.

AppEngine is awesome! Google's cloud computing platform is hosting countless web apps, from Google products to innovative personal projects, and is constantly being upgraded with cool new features. And best of all, AppEngine charges for resources (computing, storage and bandwidth) using a freemium pricing model:

"Each App Engine application can consume a certain level of computing resources for free, controlled by a set of quotas. Developers who want to grow their applications beyond these free quotas can do so by enabling billing for their application and set a daily resource budget, which will allow for the purchasing of additional resources if and when they are needed."

For example, the default free quota has a daily limit of 6.5 hours of CPU time with a maximum rate of 15 CPU-minutes/minute, while the billing enabled quota has 6.5 hours of free CPU time plus a maximum of 1,729 hours of paid CPU time with a maximum rate of 72 CPU-minutes/minute.

Billing for paid resources is controlled with a daily budget which developers set for each of their applications. The daily budget controls the amount of extra resources which may be purchased each day if the app steps over the free quota. The application developer defines how the daily budget is split between each of the billable quotas - outgoing bandwidth, incoming bandwidth, CPU time, stored data and number of recipients emailed. For example, $1 (the minimum daily budget) gets you 5 more CPU hours ($0.10 per CPU hour) and 4 more GB of outgoing bandwidth ($0.12 per GB).

Other than billable quotas, there are also unbillable quotas which increase when you switch to the paid model but are not billed. So, by switching to the paid model, irregardless to how you distribute your budget, your app gets a free bump in the:

number of requests it can process from 1,300,000 to 43,000,000 and from 7,400 to 30,000 req/min,
number of outgoing HTTP requests from 657,000 to 46,000,000 and from 3,000 to 32,000 req/min,
number of memcache calls from 8,600,000 to 96,000,000 and from 48,000 to 108,000 calls/min,

and many others. These are very usable improvements by themselves for apps that need to process a lot of requests (or bursts of requests) which don't consume a lot of CPU time, or apps that need to make a lot of outgoing HTTP requests that don't consume a lot of bandwidth.

So here's the idea for supercharging your free AppEngine quotas:

switch you app to the paid model by enabling billing,
enter the minimum daily budget ($1),
distribute the budget over resources you are 100% sure will not consume their free quota (e.g. if you have a stateless app which doesn't use the database, put the whole $1 in the stored data quota).

Since you put your budget on resources which won't consume the entire free quota and since AppEngine doesn't charge you anything if the app doesn't step over the free quota - you are essentially getting a better free quota.

EDIT (Sept 5 2009): Yesterday, a week after my original post, Google released AppEngine SDK 1.2.5 which includes a major feature a lot of developers have been waiting for - XMPP support. What's interesting and relevant for this post is that XMPP has it's own quotas for the number of API calls, data sent, recipients messaged and invitations sent, and - none of these quotas are marked as billable. Therefore, using the recipe for supercharging your free quota will also get you a significant boost in your XMPP quotas (657k -> 46000k API calls, 657k -> 46000k recipients messaged and 1k -> 100k invitations sent). Pretty cool!

EDIT (Nov 2 2009): I just noticed that it is possible to increase the limit on the number of apps each AppEngine user can create. Currently, each user may create up to 10 AppEngine apps and there was no way to create more when you reached the limit, even by paying (or was there?). However, Google engineers have been approving explicit user requests for increases in the number of apps created on the official AppEngine group. People who have requested an increase have had the limit set to 20 apps.

Are there any other hacks for squeezing more out of AppEngine? @izuzak

Named arguments in JavaScript

2009-08-09T00:00:00-07:00

While working on another project (pmrpc, which I will write about soon), the implementation of a feature came down to calling JavaScript functions with named arguments. Unfortunately, the JavaScript language (or EcmaScript to be exact) doesn't support named arguments, rather only positional arguments.

Usually, JavaScript developers get around this by defining functions to accept a single object argument which acts like a container for all other named parameters:

 1 // params is a container for other named parameters
 2 function callMe(params) {
 3   var param1 = params['param1'];
 4   var param2 = params['param2'];
 5   var param3 = params['param3'];
 6 
 7   // do something...
 8 }
 9 
10 var result = callMe({ 'param3' : 1, 'param1' : 2, 'param2' : 3 });

This solution works nicely when you are in the position to modify every function to be called this way. However, this is not always the case (e.g. when using external libraries), and also isn't as readable as having a proper parameter list, and introduces code overhead in every function that implements this principle. What I needed for my project is something that enables calling any JavaScript function with named arguments, without modifying the function itself.

What I came up with (code presented below) is based on the following:

using the toString method of the Function prototype, it is possible to retrieve a string representation of any function definition
from the string representation, it is possible to parse the number, order and names of function parameters
from a dictionary of named arguments, it is possible to construct an array of arguments in the correct order using the parsed representation
using the apply method of the Function prototype, it is possible to dynamically call a function, passing an array of arguments instead of an argument list.

 1 // calls function fn with context self and parameters defined in dictionary
 2 // namedParams
 3 function callWithNamedArgs( fn, self, namedParams ) {
 4   // get string representation of function
 5   var fnDef = fn.toString();
 6 
 7   // parse the string representation and retrieve order of parameters
 8   var argNames = fnDef.substring(fnDef.indexOf("(")+1, fnDef.indexOf(")"));
 9   argNames = (argNames === "") ? [] : argNames.split(", ");
10   var argIndexes = {};
11   for (var i=0; i<argNames.length; i++) {
12     argIndexes[argNames[i]] = i;
13   }
14 
15   // construct an array of arguments from a dictionary
16   var callParameters = [];
17   for (paramName in namedParams) {
18     if (argIndexes[paramName]) {
19       callParameters[argIndexes[paramName]] = namedParams[paramName];
20     }
21   }
22 
23   // invoke function with specified context and arguments array
24   return fn.apply(self, callParameters);
25 }
26 
27 // example function
28 function callMe(param1, param2, param3) {
29   // do something ...
30 }
31 
32 // example invocation with named parameters
33 var result = callWithNamedArgs(
34                callMe, this, { 'param3' : 1, 'param1' : 2, 'param2' : 3 });

This could also have been implemented as a Function prototype method and be called directly on a function object, which would simplify usage since the target function parameter wouldn't have to be passed. But, if the CallWithNamedArgs function indeed was implemented prototype function, could the context parameter be removed also? I.e. is it possible to retrieve the current execution context (this object) of the target function within the CallWithNamedArgs method? The problem is that the execution context within the CallWithNamedArgs method is the Function object on which the CallWithNamedArgs method was called. But what we need is the execution context of the target function, which is the Function object on which the CallWithNamedArgs method was called. That is, what we need is the current execution context of the parent object. Any ideas? @izuzak.