rhyolight.blog

It was probably four years ago that I picked up the books On Intelligence (Jeff Hawkins) and The Singularity is Near (Ray Kurzweil). Reading those two books was eye-opening. I'd never thought about the present time and its place in humanity's evolution towards the future. Although I've always been a science fiction fan, I'd never actually dreamt about our possible futures -- about the problems we'll face as a species over the next 50, 100, 1000 years. It had never occurred to me that over the past 100 years, humanity has been changing more between generations than it has historically over millennium.

Technology has increased our population to over 7 billion people, and as we begin to run out of the fossil fuels that has enabled that population explosion, we'll have to rely on technology to help us mitigate the effects of our own decline. The same technologies that have allowed us to create our dilemma provide the only tools we have to correct that situation.

Technological evolution is a natural extension of biological evolution. (I've given a presentation supporting this idea.) Therefore I have few reservations about allowing technology to evolve in most areas. Especially genetics, nanotechnology, and artificial intelligence.

I've always been interested in AI, but I missed my opportunity to become a computer scientist and study the appropriate subjects that lay the groundwork for progress in this field. I have, however, followed the progress of one company that has been doing groundbreaking research within this field: Numenta.

(download)

Click here to download:

numenta-CauGhphzhJvIygzgImFx.zip (369 KB)

Since reading On Intelligence, I have been signed up to receive Numenta's newsletter explaining their progress on "Hierarchical Temporal Memory" (HTM). Numenta is Jeff Hawkins' company established to take the ideas he defined in On Intelligence and turn them into a software product. This project is essentially a reverse engineering of how the neocortex of the human brain works.

The stuff that Numenta is working on is extremely cool. Bill Atkinson, one of the software engineers who designed the original user interface for the Mac computer, said, "What Numenta is doing is more fundamentally important to society than the personal computer and the rise of the Internet." Even recently, I've noticed some big names mentioning Numenta.

Needless to say, I'm happy to become a part of the Numenta team, and I'm more excited than ever about what I'll be working on.

Here's to changing the world.

Yesterday, Ted Dziuba wrote an acidic post about NodeJS, and there was much tweeting afterwards. I was a part of the tweeting, and I acknowledged that he has some points.

I am not a fan of the way Ted structured his critique. Calling Ryan Dahl (creator of NodeJS) a pansy was not only rude, but incorrect. I don't know Ryan myself, but he is doing something very interesting, and I'm sure he's had to fight a lot of battles to get where he's at.

And seriously, how many of us software engineers are actually doing something truly interesting?

Ted did have some valid points, especially about the event-loop synchronicity model. I think there is a bit of false advertising involved, not necessarily from Joyent and Ryan Dahl, but from the NodeJS community that has sprung up around it. If you have requirements for sufficiently parallel code execution, your code readability, maintainability, and clarity will suffer -- no matter what model you use for parallelism.

The typical NodeJS asynchronous callback approach is simple and beautiful in the easiest cases, but it gets nasty when your system gets more complicated. The problems I've needed to solve to build a framework in NodeJS have not been architecturally simpler to solve than problems in other environments.

There is no silver bullet.

NodeJS can be deceptively simple. You can find yourself dealing with the unanticipated consequences of blocking code you didn't write rather easily if you are dealing with other people's code. You can also find yourself deeply nested within closure executions because everything has to be non-blocking, and sometimes it is hard to anticipate what might block and what won't, so you wrap everything in asynch callbacks. It is easy to write spaghetti code.

Programmers should never do CPU-blocking operations on a web server, but outside a NodeJS server, the consequences of blocking are smaller. On any other web server (without the event loop idea), code that blocks will slow down the thread it is running within as well as draining CPU resources from other processes running in other threads. But in NodeJS, blocking code is more dangerous because it can halt the processing of all other requests on the server. This consequence can come from any application code running within the server, and from my standpoint as a framework engineer, it is particularly dangerous because frameworks deal with application code programmed by others. I can't make assumptions about the quality of the application code.

I like JavaScript, and I like NodeJS, but not for the event loop model. I like it because JavaScript is flexible and functional, and now I can use the same language for my applications for the server and client. I think this is the big win for SSJS. I can write web components that run on the server as well as the client. 

Saying NodeJS is cancer is purely trolling. NodeJS is brand-spanking-new. It's still 0.X.X. The community is vibrant, but still feeling out the problem space. NodeJS bucks the system. It introduces new ideas. It sparks the imagination. It stretches the limits of current technologies. All good things. I commend Ryan Dahl for his work on such an obviously successful project.

In the end, NodeJS is an interesting way to solve a problem. The wonderful thing is that we have so many choices as software engineers about how we can solve our problems.

I am currently involved in building an all-JavaScript web application framework for Yahoo! It runs on top of YUI3 & NodeJS on the server side, and the same core code also runs on the client. This "core" part of the framework absolutely cannot block the event loop under any circumstances (one of the core rules of NodeJS). This requirement is because of the server-side environment that NodeJS presents, one in which any on HTTP request handler can slow the execution of any other request simply by performing any blocking operation. A blocking operation can be an IO call to the file system, a web service, or even just a CPU-intensive task.

This is nice for the end user writing the application code because they can always assume their code is running asychronously already. It is a lot more work of the framework, however. There are very clean places the framework can call application code asynchrously, but there are also some very niche situations where it is not evident that application code is even being executed, and the framework engineers have to know these places well.

It has become a general rule for me to always assume application code blocks, and draw very large red boxes around this code as we pass it around within the framework itself. We have to ensure we always treat it consistently asynchronously no matter where it ends up executing.

It was hard to come up with a title for my YUIConf talk this year, but there it is. It looks better laid out in a slide:

The premise of this idea is to enable data to be translated into HTML either on the browser or the server. The only way to do this on the browser is with JavaScript, so we must have JavaScript on the server as well. Good thing we have NodeJS. Most web application frameworks have some type of templating engine that accepts two things: (1) data, (2) template, and returns HTML to be assembled into an HTTPResponse.

But, the richer the client gets, the more it only wants data -- not markup -- from the server. It is fairly common for an XHR response handler to expect straight JSON data and update the DOM itself, effectively taking the place of the templating engine that might do the same thing with the same data on the server.

This contradicts DRY because there is most likely server-side logic to munge that same data through a templating engine to produce HTML as well as the JavaScript function that updates the DOM with the same JSON data.

But what if we could take the browser-side JavaScript function that handles the XHR response data and run it on the server? With the nodejs-yui3 module, we can!

This presentation will specifically define our use case, then walk through an entirely open-source implementation of this idea using YUI3, NodeJS, and a Grails web server (for page assembly and dispatch).

I hope to see you there!

UPDATE: I've posted the code demo source, which is still a work-in-progress (but it does all seem to work right now).

When I was a teenager, I wanted very badly to be an independent comic book publisher. I daydreamed about it while I mowed lawns to earn extra money, and I was addicted to indie comics like Poison Elves and Cerebus. I drew things like this comedy sci-fi comic, and even interviewed with the Chicago Art Institute. I knew it was going to be extremely hard to be successful in a field like this, as many indie comics publishers will tell you if you happen to ask them.

My mother, who was much smarter than I though she was at the time, convinced me to direct my talents in other areas as well (especially ones that had decent earning potential). So eventually I abandoned the dream and replaced it with a series of misguided educational, familial, and career maneuvers that landed me enlisted in the military and sitting in front of a computer for most of my workday. This led into my career as a software engineer.

How I got from soldier to JavaScript geek at Yahoo! is not interesting. Why I Code is interesting, and I would like to talk about that. At first, I dabbled like a frightened monkey pushing unknown buttons in a lab cage, wondering what combinations divvy out the best digital treats. Soon, I could see the full potential of this amazing career field, and I wanted nothing more than to dive into it, head-first, without reservation.

But why? Why is this coding thing so awesome? Why are so many of us so passionate about what we do with our little languages that compile down into cryptic bytecode? I know Why I Code. There are two reasons.

Code is Art

There are programmers and then there are programmers. You and I are programmers, and we can have conversations about the artistic quality of beautiful code. We all know bad code because there is an abundance of it out in the world. Just stroll around github for a few hours and you'll find kilobytes of the stuff. But you and I... we know when we see really good code. I gives us a little tingle, doesn't it? You take a look at that recursive function one more time and realize why it makes absolute logical sense for it to be calling itself at that instance, and you know that's good code. It is like really seeing a painting for the first time, or listening to Music for 18 Musicians.

It breathes. You connect to it in more than just a logical way. It touches you and you can learn from it. When we write code, we express ourselves and our ideas as if the editor were our canvas and the keyboard were our palette.

Code Transcends Context

Once you grok the code, you can drive your codewagon in any direction you'd like. Our toolboxes are full of utilities with unlimited potential to launch rockets to Mars, cure cancer, map the human genome, or just map your own neighborhood. With all the power that programming gives us, we superhumans of technology can apply our skills to any field we wish.

They all need our code. They yearn for us to come and help them solve their graphing problems, their social media problems, their structural engineering problems. We need not pigeonhole ourselves into one field of work. Our power as programmers is unlimited in its application to any real world context.

We programmers have the world at our fingertips. That is Why I Code.

We are so fucking connected today. When was the last time you found yourself completely and utterly alone? No option to talk to anyone, call anyone, text anyone? No Twitter, Facebook, Google? Alone, with no one around, completely surrounded by nature, not other humans?

I spent lots of time during my youth like this, and I enjoyed it quite a bit. Now, this seems to be my nightmare. I may be lying in my bed, my computer and phone in other rooms of the house, and my brain starts pinging for connectivity. I feel like I need to at least have my phone in my hand, or on the bedstand by me. Just in case someone emails or comments on my Facebook status, or retweets something of mine. Just so I can have that instant gratification of social acceptance that drives so many of my daily actions today.

The world has change quickly and drastically, and each of us have been changed by it. 

Why?

I'm the only active developer, and I'm totally lame.

But What About My Current Grails App?

First of all, I will still monitor the Grails user mailing list for questions about GrailsUI and reply as I have time to help out.

Also, there may be some GrailsUI users out there thinking they made the wrong decision about which RIA plugin to install, but read below before you get upset with yourself (or me). The bottom line is this: If you are making a considerably complex RIA application using JavaScript, you will be better off hand-coding it with JavaScript than using a plugin as an RIA platform.

My Thoughts About GrailsUI

Initial roles of GrailsUI:

1. Simple AJAX widget provider
2. Platform for RIA frontend

GrailsUI does #1 pretty well, in that it abstracts the complexity of creating AJAX widgets away from the Grails developer. You don't have to write any JavaScript to get a basic widget working, but if necessary you can write some JavaScript to interact with the widget on the page, thus adding functionality.

Although it seems like GrailsUI does #2 well as well, this becomes largely unmaintainable as the codebase increases, and especially as the importance of performance increases.

It turns out that GrailsUI is a horrible thing to build an entire RIA codebase on. In fact, if the design goals of a RIA front-end are extensive, I believe the code should be written entirely by hand.

It is just like putting down the WYSIWYG editor and hand-coding your UI. GrailsUI might provide a great way to layout some basic AJAX stuff, but as soon as you get into any non-trivial JavaScript work, it breaks down in an unmaintainable heap.

I do not think the concept of an RIA platform plugin for Grails is a bad idea, however. I just think that I lacked the wisdom and foresight to create that platform with GrailsUI.

There are several maintainability issues within the JavaScript and markup that GrailsUI creates, namely:

• Global JavaScript variables. They are all over the place. I would not be surprised if a lot of the GrailsUI bugs are because of this.
• JavaScript liberally sprinkled throughout the markup, and generated in a very dynamic way. This really complicates things.
• Scripts all load at the top of the page.
• Core GrailsUI JS lib augments native JavaScript object (String)
• JSON responses are manually eval()'ed instead of going through a proper JSON parser
• There are 0 JavaScript tests, and minimal Groovy tests

It is entirely possible to address each of these issues in a rewrite of GrailsUI, and in fact I was planning on doing this with GrailsUI 2.0. But this is a major task.

All that being said, there are a lot of GrailsUI users, and many of them have thanked me for this plugin, saying it has made their lives easier. Like I said, I think a plugin like this has a place in the Grails ecosystem, but I think it's primary role should be that of a shortcut to AJAX widget goodness, no so much shortcut to an AJAX platform. It can also be useful as a stop-gap measure to get a UI in place before major front-end resources are brought in to cleanup and re-implement.

Thank you

Thanks to all of you who use GrailsUI, for making it one of the most popular Grails plugins. Especially thank all of you who submitted bugs and attached patches, and especially Daniel Honig, who was instrumental in getting the first version of GrailsUI off the ground.

Would You Like to Take the Torch?

Awesome. I've already established a group for it with a planning page for GrailsUI 2.0, and there are actually members. This message is getting posted there as well. What the GrailsUI project really needs is a leader with a good amount of JavaScript and Grails experience -- and the motivation and time to dedicate to an open source project.

In Summary

This is one of those "if I knew then what I know now" situations. When I started GrailsUI, I had very little experience with JavaScript, so I did a lot things wrong. Now that I know JavaScript really well, I cringe when I look over the GrailsUI codebase. I would love to take six months and completely re-write GrailsUI, and if anyone is willing to pay me to do so, I would be happy to. But now that I'm working at Yahoo!, my plate is full of JavaScript work already, and I am choosing to use my spare time for other endeavers. 

If you would like to contribute, please contact me or join the dev group.

_{Also, here is the Grails user mailing list discussion on this topic.}

I assumed there might be some differences in performance between the classic prototypal inheritance object construction and the explicit object construction models, so I wrote some code that tested it using the models below:

Classic

Object

I ran each constructor function 1,000,000 times in Chrome and tallied up the resulting times it took for each construction, with some interesting results:

With no inheritance at all, classic object construction is over 8 times faster than the explicit object construction method. On constructing an object with one superclass, classic construction is over 36 times faster. And on constructing an object with two superclasses, over 46 times faster. These are pretty telling results, but you must keep in mind that I've created one million objects in each case. 

These results are from Chrome, but I also ran them through Firefox. The results gave the the same conclusions, but object construction in Firefox is roughly 10 times slower than Chrome across the board. The remaining examples are all from Chrome, just because it was so much faster to test.

These results have me an idea to test a deeper inheritance structure, so I put together some code (not shown here, but contact me if you are interested) that created an inheritance hierarchy that was 10 levels deep. The first case was just an empty model with no attributes, but a toString method that was overridden in each subclass to return a different string. In the second case, I added only one attribute to the topmost superclass. For this case, in the object-construction model, I also added the typical getter and setter methods to the topmost class as well. Here are the results of creating 1,000,000 objects of each level of inheritance with both the classic and object-explicit construction models.

The flat lines on the bottom are representing the classic model, one without any attributes, the other with one attribute. As you can see, there is a linear growth in construction time through all the models, but the slope of the explicit object construction model is much higher, especially when there is an attribute involved even at the topmost inheritance level.

I didn't take the inheritance heirarchy past 10 generations, but if you look at the data another way, you can see that the discrepancy in construction time starts to level off at around 50 times faster at 10 generations.

Here are some conclusions I got from running these tests:

• Chrome is over 10X faster than Firefox at JavaScript object construction.
• If you forget the 'new' keyword using the class construction model, you are screwed. You must use 'new', and there is no error thrown if you forget about it. The function just executes without the extra "construction" features.
• If you forget the 'new' keyword using the explicit object construction model, you actually get a small (negligible) performance boost.
• Large inheritance hierarchies should be avoided no matter with method you use.
  • In both cases because it is just friggin' confusing
  • In the explicit object model because it can get really slow
• If you must have a non-trivial inheritance hierarchy, you need to know approximately how it will be used. If there will be many instances created of the model, maybe the classic inheritance scheme is preferred for performance reasons.

Lately, I talked about creating classes of objects in JavaScript by having the constructor function explicitly return an object literal. Using this method, the resulting object has no prototype, and the constructor does not need to be executed using the 'new' keyword (although it is a good idea to use the 'new' keyword in order to explicitly specify object creation to other programmers). Now, we're going to talk about creating a local extension of this "class" of objects. We ended up with this constructor function:

And now we want to create a local extension to this called PositionedRectangle, which adds location data and a new method to calculate a distance between itself and another PositionedRectangle. With the first class example of prototypal inheritance, we created a new constructor function that called the "superclass" constructor, then did much mucking about with the prototypes of the two constructor functions in order to hook them together into an inheritance model. While we still have some mucking about to do in the form of encapsulation, but we can accomplish this without too much trouble. This inheritance model will create a new object of the type of the superclass, then use it to create a new object that contains its properties in addition to the properties the subclass defines. It looks like this:

This constructor function is a local extension of the Rectangle constructor above. It accepts parameters representing width, height, x position, and y position. It also creates a new object using the Rectangle constructor, which we use later. The last thing it does is create a new object literal that contains all the properties we want to be available in PositionedRectangle objects in addition to those that come along with the Rectangle constructor function. There is a 'mixin' method that somehow magically puts the instance of Rectangle and the definition of our PositionedRectangle together, and that is what is returned. The input values are assigned to local function-scope variables that can be accessed by the methods declared within the 'me' object literal that is used in the mixin function.

Essentially, we create two objects using the input parameters to the constructor, then we mix the two objects together to create something new, and that is our result. The first object is an encapsulated instance of Rectangle, which we'll use for the mixin and possibly local overriddings of methods (we'll do that in a minute). The second object contains a collection of properties that belong to a PositionedRectangle instance, including accessor methods and also a distanceFrom method that accepts another PositionedRectangle and calculates the distance from it. Time to take a look at the magical 'mixin' function:

In our case, the 'mixin' function is looping through all the properties of the Rectangle instance we created and putting them directly into the new PositionedRectangle instance we are creating. The only condition is that the property cannot already exist in the PositionedRectangle instance. This is just one way of going about it, but I like it because it allows me to override methods of the Rectangle, as I'm doing below:

Notice the 'toString' method attached to the object under construction above. In order to take advantage of the toString method within Rectangle, it uses the reference it has to the encapsulated Rectangle object that was created for the mixin process, then adds the x/y location to the string. So now:

Produces this:

(2X3)

6

(6X3) at 0,0

18

141.4213562373095

Our new instance of PositionedRectangle still has reference to all the Rectangle methods, and its own methods. And it overrides a Rectangle method while still able to call the super class method from within. I'll try to make another blog post about the performance implications of each of these inheritance methods soon.