Archive for the ‘JavaScript’ Tag

SQL, JSON and embedded JSON

No answers today, only questions.

My application is using JSON objects to store data. Initially I didn’t like them because they felt kind of sloppy; you didn’t know if the things inside were defined, were arrays, hashes, functions or what. Now, I really like this flexibility (and, yes, I know JS doesn’t have a hash).

So, how do I store this complex object in my database? So far, it hasn’t been much of an issue. The object is a list, the list has values in JSON and so the object is a table in SQL and the values are records. No big deal.

But, now I have more complex objects. They are essentially tables within tables within tables. So, the translation code can get extensive. And, I have way too many joins.

One thing I tried is to Base64 encode the JSON object and just stick it in the database. This actually works pretty well, except that I can’t get to the data with a simple SQL query. I first have to get the blob, decode it, convert it into something useable and then do processing on that. Conceptually, no so hard, but a huge pain when much of your data is hidden away like this.

My answer for now is to simplify my application so I don’t have this problem. But, it isn’t gone, just deferred. If anyone has advice, I would love to hear it.


My Javascript Object maker

While using my solution to clear, easy to understand and short asynchronous Javascript, I ran into a big problem. I wrote it properly(1), with private variables and all. But, if another part of code called it, the variables would collide. It was just a function. Obvious in retrospect, like most problems!

I need an instance of the function for each call. But, I really don’t like “objects.” I’m not sure why. So, I decided to write an “Object Maker” that will accept a function as an input.

Here it is:

PX.asyncObject = function (spec, my) {
    var that = {};  // declare PRIVATE variables here
    my = my || {};  // Use my.xx for *shared* variables
    if (spec.objName) {
        eval('var ' + spec.objName);    // GLOBAL vars are *more* evil than eval
        window[spec.objName] = spec.newObj;             // Make the function sent into a method named from objName
        that[spec.objName]   = window[spec.objName];    // Make the new method a method of "that"
    } else {
        var go;
        go      = spec.newObj;  // Make the function into a method named "go"
        that.go = go;  // Make "go" a method of "that"
    that.spec = spec;    // Put spec into private vars. call go({vars})
    return that;  // return the new, private, durable object

The basic routine and idea came from (surprise) Douglas Crockford’s “Javascript: The Good Parts,” pp. 52-55. It is called a functional constructor.

I use it like this:

    newObj: function (spec) {
        PX.Rest('GET', sUrl, undefined,
            function (result, o) {
                if (result) {                         
                    // Code for success
                } else {
                   // Code for failure
                    // Code for both

Four lines longer and only a little less readable.

(1) Using a Module (pp. 40-41) from Douglas Crockfords “Javascript: The Good Parts.”

My Javascript Package Manager (again)

I just finished re-writing the “loader” piece of my package manager. It turns out that I was using a function instead of an instance of a function. So, when I thought everything was working and tried loading three packages all at once: ka-blam! They stomped all over each other (as should have been expected, in retrospect).

So, I created an “instance creator” routine. This way, I can take one of my functions and send it as a variable to the instance creator and get back an instance of the same function. Since I’m sure I’ll have this problem frequently. And, since I hate writing routines that are real objects to begin with (not sure why; maybe I’ll get over it some day), I now have a generic solution.

My “loader” can now, from the input of a package name and a function (e.g. init or open), download all specified files except those already loaded, make the necessary queries except those that the data is already loaded for, then render the template once everything else is in place.

I’m hoping to share some code as soon as it is reasonable.

Package Manager File Loader

The package system I originally came up with is getting more complex by the day. First, I wanted to make sure I could load files.

YAHOO.util.Get solved that easily.

Then I wanted to be able to minify and compress them.

YUI Compressor took care of that.

Finally, I wanted to be able to combine them into single files. Again, not really hard at all. In fact, I just wrote a script that essentially does “cat file1 file2 file3 > combofile”.

So, now I have the possibility of:




(I had /js/corefiles-min-combo.js for a while too, but then realized if I wanted it minified, I just needed to make the combo definition include minified files to begin with). See, sometimes things get more simple!

Next, I need to make sure that I only download a file once. So, when I want to dynamically load a new file, I need to make sure I don’t have the plain version, a minified version or either one of those as part of a combo file.

I’m only part of the way through this code. No YAHOO.package.manager to save me this time (if I missed it somehow, please let me know!).

So far, I can list all loaded files. I can then strip the “-min” off (since, from the program’s point-of-view, it doesn’t matter if the file is min’ed or not). I can also take a -combo file, check the package definitions for what is in that combo file,  then add those files (again, removing the -min) to the list.

Now, I have a “master” list of all loaded files, which I can check before trying to load a file.

Next up is to determine if it is faster to add files to this array as I load them or to simply rebuild the entire list every time I load a new package. It seems safer to just rebuild, but sloppy. Keeping track means things could sneak in if a file gets loaded and doesn’t “register” itself with the list tracker.

We’ll have to wait and see what I come up with.

My Synchronous Javascript Solution

I wrote about the problem I was having with the asynchronous (albeit single-threaded) nature of Javascript in my last post.

After looking at the options to make my code more clear when using asynchronous calls in Javascript, I’ve decided to take a simple approach. Instead of using one of the libraries, I’ve come up with what is basically a different way to format my code.

I can’t tell if this will be my long-term answer yet; I’ll just have to get further into the project and see how it goes.

So, here’s what I did. First, I wrote a function to do the actual asynchronous call. The only special thing about it is that it returns TRUE or FALSE, plus the return object.

PX.Rest = function (method, url, data, callback) {
    var successHandler, failureHandler, makeRequest, request;
    successHandler = function (o) {
        callback(true, o);
    failureHandler = function (o) {
        callback(false, o);

    makeRequest = function () {
        YAHOO.util.Connect.initHeader('Content-Type', 'text/json', true);
        request = YAHOO.util.Connect.asyncRequest(
            { success: successHandler,
              failure: failureHandler,
              connTimeout: 8000

Now, this lets me write my code like so:

// The first AJAX call.
PX.Rest(method, sUrl, jsonData,
    function (result, o) {
        if (result) {
            // If I'm here, it means the initial call was successful.

            // The second AJAX call (for example, to tag the record I just created).
            PX.Rest(method, sUrl, undefined,
                function (result, o) {
                    if (result) {
                        // The second call passed also
                    } else {
                        // The second call failed (after the first call succeeded)
                    // Code that runs no matter if the second call passes or fails
            // Code that runs when the first call passes but BEFORE the second one returns.
        } else {
            // The first call failed.
            // In this example, I don't do anything else.

Now it flows!  It’s almost like using nested if statements.

Here’s why I (think) it works:

  • The arguments are ordered in such a way  that the last one is the callback. This allows you to simply indent it in order to make it obvious you are inside conditional code.
  • The PX.Rest function from above returns TRUE or FALSE, which allows me to use a ‘If (request)’ in-line. That way, I can continue to simply indent and don’t have to create separate callbacks, define them ahead of time, etc.

I certainly don’t think this is a perfect solution, but it does allow me follow the flow of my code much more easily. I wouldn’t want to go more than two or three calls deep, though!

Wrapping my head around Asynchronous Javascript

Most of the Javascript I use involves AJAX. Even before I really knew how to program in Javascript , my first cut & paste programs were AJAX simply because of the site I’m building. As I get deeper into the coding, trying to get multiple parts of my site to work together, the asyncronous model is getting harder and harder to work with.

Here is a simple pseudo-code example:

myFunction = function () {
    successHandler = function () {
    failureHandler = function () {
    ajaxSetup = function () {

That’s not too bad. As I’ve said before, I think it’s a bit verbose, but workable.

Putting it into the regular flow of code is harder. Here is my (made up) example:

  1. Create a new window
  2. Get data from the server
  3. Format the data with links
  4. Update a counter on the server

In pesudo-code again, it might look like this:

myFunction = function () {
    successHandler = function () {
        myCounterUpdate = function () {
            myCounterUpdateSuccessHandler = function () {
            myCounterUpdateFailureHandler = function () {
            myCounterUpdateajaxSetup = function () {
    failureHandler = function () {
    ajaxSetup = function () {

Yuck! I actually have some code like this and was just going to deal with it. But, one of my flows required a third layer.

I simply can’t maintain any sense of the program with so many layers! So, I went looking for solutions and found this blog post by Harry Fuecks. It’s a couple of years old, but seems to capture exactly what my issue is and has some solutions. One is called Narrative Javascript and another is jwacs.

I’m not sure I like any of the solutions, though, since they involve adding a lot more code. But, I’m still wrestling with this issue, trying to re-factor a bit to gain control and see what other solutions there might be.

Managing a non-trivial amount of JavaScript

Even though I’m still new to JavaScript, I’m using it most of the time now. My initial attempts with copy & pasting code, watching YouTube videos, reading blogs, etc. were not getting me up to speed fast enough. I also came to realize that I had some fundamental misunderstanding about the language. So, I took a break and read Douglas Crockford’s JavaScritpt: The Good Parts. How refreshing!

Now many of the  issues I was having, especially when comparing Perl code with JavaScript, had real answers. And, getting some real answers for differences in style of JavaScript code had some actual answers as well (due to things like the automatic semi-colon insertion).

My biggest discovery of all is the “Method Pattern.” (pp. 40-21). Eric Miraglia explains it very, very well in his blog post. This shows how to avoid Global Variable problems, make code more modular and all that other good stuff. It is *very* verbose (in my opinion). But, it works, I understand it and I can build a site with code structured like this with confidence.

With this new knowledge and the knowledge I already had about page size and performance issues with Rich Internet Applications (RIAs), I developed a way to load code on demand. I first wrote a “Loader” script. Then, I wrote a template to use that would interact with the Loader.

An example. Let’s say I wanted to open an interactive window. This window might require CSS files, JavaScript files, markup and data. And, some of it might already be loaded, but there is no guarantee. So, instead of dealing with this when you want to make the call, just call the Loader script with the action you want and let it take care of it.
To open a dialog box:"dialog_box");

To close the dialog box:


To make this work, I would need a standard set of commands (init, open, close, etc.)

So, if you want to check it out, here is the actual code. I ran it through JSLint with Recommended Options and Good Parts turned on. Be aware that I’ve only used it a bit and so have not flushed out all the issues that will make this truly robust.

Note: px is just a placeholder name. “plain_jane” would become “dialog_box” or “category_dialog) or whatever feature you were implementing. Oh, and I log. A lot.

// Copyright (c) 2009 Brian Estes - All Rights Reserved

/*global YAHOO */
YAHOO.px.plain_jane = function () {
    YAHOO.log("Instantiating", "begin", "pxPlainJane");

    var example, setupListeners;

    example = function (o) {
        YAHOO.log("examle", "begin", "pxPlainJane");

        YAHOO.log("example", "end", "pxPlainJane");

    setupListeners = function () {
        YAHOO.log("setupListeners", "begin", "pxPlainJane");
            function () {
                YAHOO.log("clicked plain_jane_update", "info", "pxPlainJane");

            function () {
                YAHOO.log("clicked plain_jane_one", "info", "pxPlainJane");
        YAHOO.log("setupListeners", "end", "pxPlainJane");

    return {        // public methods
        init: function (o) {
            YAHOO.log(".init", "begin", "pxPlainJane");
            YAHOO.log(".init", "end", "pxPlainJane");
        open: function (o) {
            YAHOO.log(".open", "info", "pxPlainJane");
        refresh: function (o) {
            YAHOO.log(".refresh", "info", "pxPlainJane");
        close: function (o) {
            YAHOO.log(".close", "info", "pxPlainJane");
        destroy: function (o) {
            YAHOO.log(".destroy", "info", "pxPlainJane");
        exists: function (o) {
            YAHOO.log(".exists", "info", "pxPlainJane");
        visible: function (o) {
            YAHOO.log(".visible", "info", "pxPlainJane");
        ready: function (o) {
            YAHOO.log(".ready", "info", "pxPlainJane");

YAHOO.util.Event.onDOMReady(function () {

YAHOO.log("Instantiating", "end", "pxPlainJane");

Make and Combine

The build script is working pretty well now. I decided to split the build and the combine scripts into two. It turns out that they were essentially two sets of code anyway and the only thing they had in common was that you run combine after make most of the time.

There are still issues knowing which scripts have loaded. If I include a script inside a combo file, YUI’s Get (or the browser) doesn’t know if it is about to pull in a duplicate set of code. Since I have the list of files inside the combined file, I can check to see if each file is loaded before passing it off to Get.

I’m also a bit concerned that I have integrated this too deeply. To have server side and client side scripts both using the same configuration file is great at preventing you from forgetting to make changes in two places. But, the implications of a change need to be considered across two different code-bases.


Everything is working today, although I would not consider this script done. It will minify and compress all of my JavaScript and CSS files, compile the Jemplates and then build concatenated versions of any files I specify. It is smart enough to know that files that end in -min or -combo are “object” files and not to delete them.

I’ve also move the configuration for the package (“combo”) management system into a separate file that both the build script and the core JavaScript file pull from.

The issues I have now: how to deal with duplicate copies of the same code if it is in regular, -min or -combo. I have a list of all the files in the combo files, but there is still a chance to duplicate one if it is the “debug” version and the other is not.

Also, I haven’t coded the part the knows to skip files that are already loaded as part of a combo. I’m using YAHOO’s Get feature to download files (and it is smart enough to know if a file has already been downloaded), but I’m having to duplicate some of this functionality myself, which is not a good thing.

This is taking much longer than I want. But, seeing a single file that contains all of my current JavaScript & templates load which is smaller than just ONE of the files I had before does show that this is really going to make my site much faster.

Next up, image sprites.

Again with the combo files

With a trip to San Francisco and a Walk-a-Thon at my kids school today, it is really cutting into my productivity. Or, at least it’s a good excuse; I can’t tell which.

I looked more carefully at YUI Builder today and the ant-driven procedure they use to build YUI itself. I probably really should be using something like this, but since I don’t totally understand it, I’m reluctant.

I did spent quite a bit of time first refining my build script and then adding the combo file functionality. It actually works right now, but I’m going to take another look at it after a little brake to make sure it is doing exactly what I want.

Because I’ve forged a deep relationship between my build and my package (“combo”) management system, there are more cases to consider.