quiet light falling

Trial and Error and the God Complex...

2011-07-17T19:53:00.003-04:00

I just saw Tim Harford's TED talk: Trial and Error and the God Complex.

Harford makes some good arguments for evolutionary, "trial and error" approaches to solving complex problems. There are also significant risks.

Harford's examples are very carefully chosen not to present the limitations of evolutionary approaches to solve problems. For example, a human baby is a product of evolution, but without a specific design -- a product of natural environmental constraint and common decent. Harford's "nozzle" story is different, there is a desired design, which is found through a genetic algorithm... in this case, the principle of evolution is applied in an artificial environment where the constraint is a metric on a fitness function (e.g. how well does the nozzle make detergent?) This allows the genetic algorithm to incrementally improve the design without actually knowing how the design works, by passing traits that contribute to passing the fitness function.

It is helpful to realize that top-down design and genetic algorithm design are simply different ways of finding solutions (optima). One isn't better than the other, any more than analytic solutions to functions are better than numerical solutions, nor are they always equivalently easy or possible to find.

Genetic algorithms only work for designs when a fitness metric can be established (e.g. how well the nozzle makes detergent) -- it is not useful or appropriate as a technique when a fitness function is unknown. Hence, you already have to have expended considerable thought to define a fitness function that correctly specifies the question in order to solve problems the way Harford suggests. This isn't trivial in the general case, it can be harder than solving the problem using top-down design methods.

Harford's example of the stock market is not completely unrealistic -- it is entirely likely that "something" is evolving faster than humans -- but since we have only very vague ideas about what effects a fitness function like "fiduciary trust" has, we don't know exactly what corporations will evolve to claim us as pets.

This brings us to the second shortcoming of evolutionary techniques.... while they are excellent at homing in on optima using a fitness function, the only thing you can say about them is the probability that they are good enough to pass the fitness function and nothing more. If your fitness function forgets to include exceptions, there is no guarantee that your solution will remain stable for such conditions. Even more fundamentally, you have no idea how the solution actually works.

Genetic algorithms can lead scientists to breakthroughs, however in order to understand how they work, the solutions must be reverse-engineered back into an analytic understanding.

Harford focuses so closely on the amazing ability of these systems to find answers that he has forgotten that those answers need rigorous testing. It turns out that humans are even worse at defining correct testing scenarios than we are at building systems top-down. So while his approach may be compelling, it is not without significant danger. It substitutes one evil (i.e. what he calls the "God complex") with another: the hubris to think that we know what we want (i.e. understand the fitness function) sufficiently to know when we get it.

Asimov warned us of this hubris in stories featuring the notorious well-meaning "Laws of Robotics" that led robots to very different conclusions than expected. And Crichton warned us of trying to control such systems as they rapidly change and exceed our ability to control them.

Real evolution is raw, untamed, and unpredictable -- it doesn't play by rules we fully comprehend.

Good Lesson, but not universal...

2011-07-11T15:58:00.002-04:00

Aaron Iba explains in his blog that his iPhone game led him to a new way of thinking about development that he thinks makes him a better programmer and startup founder.

"Iterate-and-Repair" is a good optimization strategy where opportunity costs are low (i.e. trying out different solutions in a puzzle game), but they aren't so great where opportunity costs are high (i.e. trying to send a man to the moon). In that case, the best strategy is still to engineer an optimal solution using mathematics.

The key distinction is not learning styles, nor even an MIT background, but rather asking the question: "what is the opportunity cost for my solution?"

The late 90's saw a wave of "good enough" engineering that was based on a similar idea of iterate-and-repair, but notoriously did not factor opportunity cost in the analysis. Who needs all those edge-cases and testing when you can follow the 80-20 pareto rule and get to market quickly? Well, if you've ignored the opportunity costs and your solution fries a thousand people in an aircraft, or tanks a billion dollars in stocks overnight, you better believe that an 80/20 approach is not "good enough" after all.

Of course you should use the right tool for the job, but there is still a need for engineered solutions.

ruby and json vs xml marshalling

2011-02-17T14:19:00.006-05:00

I've been watching a silent war rage between the ruby community with its preference for JSON and RESTful services and the Java community with its preference for XML and SOAP.

The rdoc for Nokogiri snarks at the Java community with a simple quote:

"XML is like violence - if it doesn’t solve your problems, you are not using enough of it."

Like many devs caught in the middle of this war (namely having to integrate Rails apps with Java SOA backends) I'm never quite satisfied with the state of Ruby XML marshaling. Sure there are a dozen gems to address this -- they always vary between giant thousand-object DOMs that leak memory like a sieve, arcane forced DSL syntaxes, or simple to use, but incomplete APIs. So I started to think about hand rolling "yet another XML marshaller" for Ruby.

My first idea was to simplify the XML DOM, so that everything is unmarshaled as an element. Then I can use a simple hierarchal accessor, the "dot" operator on the Ruby object side:

On the face of it, I thought that looked pretty sweet. But then I considered remarshaling this object as XML. Doh! There's no way to tell whether "mike" is an element or an attribute... well I could stick a little metadata on there to remind me later, but it's starting to look like I stumbled into the same path that so many other Ruby gems had trying to solve this problem. And it's not for lack of trying -- Ruby devs have tried many ways to resolve this ambiguity with various results, but not a lot of clear wins -- there's always a "gotcha" in there that makes you wince when you have to handle it. (it's the same wince that Java devs have when they switch axis jars on one server but not another.)

I sat back in disgust and thought to myself,

"wow, this would be so much easier if it was just JSON..."

then it hit me... JSON is easy because round-trip marshaling doesn't introduce any ambiguity about elements vs attributes! Everything is an element!!!

No wonder JSON is so easy!

How to get Surface 1.0 SP1 running on Win7 x64 and VS2010

2011-02-05T19:23:00.003-05:00

I'm posting this "brain dump" from my research journal because it was a royal PITA. Maybe it can help someone else. The articles referenced in here hopscotch around in quite a tangled fashion. I empathize with the people who wrote these up... we all have the tendency to document the things we know about at the time we know them and then quickly go stale or have quirks. This post will be no different, but at least as of 2/5/2011, this tells devs new to Microsoft Surface 1.0 SP1 development how to proceed. I hope it can save some grief.

Twitter thanks to @GrumpyDev, @eNeRGy164, @surface, @sevensteps, @brianpeek, my friend @gaganrajpal, and Prof Holly Yango for her quickstart (http://www.cs.uml.edu/~holly/)

back to work. going to try to install the SDK again and get it working.

previously I had used Brian Peek’s instructions at http://www.brianpeek.com/blog/archive/2009/03/10/install-the-surface-sdk-on-windows-7-and-or-x64.aspx but there are some slight variations here. @sevensteps advised that grumpydev was worked.

BULLETS mark steps I did.

from my 1/27 post:
GrumpyDev was identified as should work if I follow it to the letter, will try this weekend again: http://www.grumpydev.com/2009/05/17/surface-sdk-sp1-on-vista-and-win7-x64/

download MSI: http://www.microsoft.com/downloads/en/details.aspx?FamilyID=3db8987b-47c8-46ca-aafb-9c3b36f43bcc&displaylang=en

use ORCA to remove the following keys from the MSI.

Select “LaunchCondition” in the left hand list, then select “Install OR NOT VersionNT64″. delete
select “InstallExecuteSequence” in the list on the left and remove SetCreateSqmMachineGuid and CreateSqmMachineGuid. delete

Followed this to the right after it says save and close ORCA part, but before installing flip over to these instructions to get orca edits for VS2010 as well:
http://blog.hompus.nl/2010/04/14/using-the-surface-sdk-with-visual-studio-2010/

Select “LaunchCondition” and:

Select the row with "Installed OR (VS2008SPLEVEL AND VS2008CSPROJSUPPORT) OR VCSEXP2008SPLEVEL" and choose "Drop row"
Select the row with "Installed OR (VS2008SPLEVEL AND VS2008SPLEVEL >= "#0") OR (VCSEXP2008SPLEVEL AND VCSEXP2008SPLEVEL >= "#0")" and choose "Drop row"
Select the row with "Installed OR DEXPLORE" and choose "Drop row"
Select the row with "Installed OR VS90DEVENV OR NOT VS2008SPLEVEL" and choose "Drop row"
Select the row with "Installed OR VCSHARP90EXPRESS OR NOT VCSEXP2008SPLEVEL" and choose "Drop row"

Now, continue from [http://www.grumpydev.com/2009/05/17/surface-sdk-sp1-on-vista-and-win7-x64/] and:

Now you can quit Orca and run the msi, which should install just fine. Make sure you also switch off automatic updates and error reporting as follows...
done

Ok, the SDK is installed. Now, following the instructions it says to go to the original post and follow steps 2 & 3 (http://www.grumpydev.com/2008/12/26/surface-sdk-on-vista-x64/):

me:

changed step 1 to “open command prompt with Run as Admin”

cd “C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC”
vcvarsall.bat

then:

Change to "C:\Program Files (x86)\Microsoft SDKs\Surface\v1.0\Tools\Simulator".
Enter "corflags SurfaceSimulator.exe /32bit+ /force" (without the quotes). We need to use /force because the assemblies are strong named signed, and altering them this way will invalidate that signature. This is unfortunately unavoidable, but I haven’t found any problems from doing so. You will receive a CF011 warning informing you of this
Change to "C:\Program Files (x86)\Microsoft Surface\v1.0".
Use the same corflags syntax from step 3 on the following files, one by one:Attract.exe, AttractConfig.exe, SurfaceInput.exe, SurfaceOutOfOrder.exe,SurfaceShell.exe.
Close the command prompt and launch the simulator!

finished step 2. simulator appears to run. however closing it doesn’t shut it down. I had to go to the task explorer and kill the process tree for the simulator. still, this is a step in the right direction.

Now, on to step 3. but wait, I don’t have any projects yet. Ok, we’ll come back to this.

Going back to http://www.grumpydev.com/2009/05/17/surface-sdk-sp1-on-vista-and-win7-x64/ he explains in an update how to install the samples.

I had to do this:

goto C:\Program Files (x86)\Microsoft SDKs\Surface\v1.0\Samples and unzip “Surface Code Samples.zip” into the same dir.

had to expand this to a different folder.

now follow his update:

Edit the configuration properties for the sample solution and create an x86 platform for all projects. (this is step3, haven’t done it yet)
- Edit InstallSamples.bat
- Comment out the lines on either side of the set as follows:
::FOR /F “eol=H tokens=2*” %%A IN (‘REG QUERY HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Surface\v1.0 /v IsLogicalSurfaceUnit’) DO SET LogicalTableValue=%%B
SET InstallingOnTable=false
::IF %LogicalTableValue%==0×1 SET InstallingOnTable=true
- Change the MSBuildParameters to reference the x86 PlatformName as follows:
SET MSBuildParameters=/p:Configuration=Release;PlatformName=x86 /noconsolelogger /fl /fileLoggerParameters:LogFile=%LogFile%;Append /nologo

ok, now I did:

run cmd as admin

cd “C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC”
vcvarsall.bat

cd (where I unzipped the files)
InstallSamples.bat

“Successfully installed...”
“Please run the Surface Simulator to browse sample applications from the Surface Shell Application launcher”

Hmmm, wonder what that is? oh!! just running the simulator now has a bunch of apps in it. Ok, there must be a directory or database where the samples got “installed” to in order to run in the simulator...

Hmm however none of the samples actually run when I open them -- timeout. Hmmm.

Wait, maybe the XNA stuff isn’t installed?
Reading this: http://blog.hompus.nl/2010/03/03/installing-the-microsoft-surface-sdk-on-windows-7-x64/

download and install XNA 2.0 redist: http://www.microsoft.com/downloads/en/details.aspx?familyid=15FB9169-4A25-4DCA-BF40-9C497568F102&displaylang=en

Nope, I had already done this, tried uninstalling and reinstalling. no dice.

Ok, what about this? I noticed that grumpydev http://www.grumpydev.com/2008/12/26/surface-sdk-on-vista-x64/ step 2, substep 5 was missing this file:
setupcustomaction.exe
this is mentioned at http://blog.hompus.nl/2010/03/03/installing-the-microsoft-surface-sdk-on-windows-7-x64/
so I’m doing this from that dir:

CorFlags setupcustomaction.exe /32BIT+ /Force /nologo

Still no dice... everytime I launch a sample, it starts up, I click something and get “ControlsBox has stopped working” -- or similar depending on the sample.

Ok, forget the samples. Let’s go back to the instructions for installing the solution templates:
http://blog.hompus.nl/2010/04/14/using-the-surface-sdk-with-visual-studio-2010/

COPYING THE TEMPLATES FROM THE SDK:
xcopy /s "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\ItemTemplates\CSharp\Surface\v1.0" "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\ItemTemplates\CSharp\Surface\1033\"
xcopy /s "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\ProjectTemplates\CSharp\Surface\v1.0" "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\ProjectTemplates\CSharp\Surface\1033\"

ok, that was bunk. those dirs don’t exist on my system. I think he meant:

C:\Program Files (x86)\Microsoft Visual Studio 9.0\Common7\IDE\VCSExpress\ItemTemplates\CSharp\Surface\v1.0

WpfSurfaceCustomControl.zip

WpfSurfaceUserControl.zip

WpfSurfaceWindow.zip

WpfTagVisualization.zip

which did exist... but hmmm, there is also...

C:\Program Files (x86)\Microsoft SDKs\Surface\v1.0\Project Templates

WpfSurfaceApplication.zip

XnaGS3SurfaceApplication.zip

XnaSurfaceApplication.zip

SO CONFUSING!!

Wait, hold on, the target directory already exists too!!:

C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\ProjectTemplates\CSharp\Surface\1033

WpfSurfaceApplication.zip

XnaSurfaceApplication.zip

each of these dirs has different files. GOOD GRIEF!!

Ok, let’s just try running the last two commands and see if VS2010 recognizes the two templates installed in VS 10.0 dir above.

cd "C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE"
devenv /setup

Ok, after it completed it picked up the two installed templates in the VS2010. Good! To see this:

Open VS2010 and go to “New Project”
expand “Other Languages/Visual C#/Surface”. See:

“Surface Application (WPF)”
“Surface Application (XNA)”

NOTE: http://msdn.microsoft.com/en-us/library/ee804897(v=surface.10).aspx says:
The Microsoft Windows SDK for Windows Server 2008 and .NET Framework 3.5 is not required. However, if you want to develop Microsoft Surface applications by using Windows APIs, you must install the Windows SDK for Windows Server 2008 and .NET Framework 3.5.
(I already had installed Windows SDK (not server 2008) and .NET 3.5 previously).

ALSO NOTE from the same loc:
The Microsoft Surface SDK does include project templates for XNA 3.0, but it does not include XNA 3.0 components. If you want to develop Microsoft Surface applications by using XNA 3.0, install Microsoft XNA Game Studio 3.0. When you want to deploy a Microsoft Surface application that uses XNA 3.0, install Microsoft XNA Framework Redistributable 3.0 on all Microsoft Surface units that will run the application.
(I had installed the XNA Game Studio 3.0 previously).

http://www.microsoft.com/downloads/en/details.aspx?FamilyId=7D70D6ED-1EDD-4852-9883-9A33C0AD8FEE&displaylang=en

Note that http://blog.hompus.nl/2010/04/14/using-the-surface-sdk-with-visual-studio-2010/ also states you MUST select .NET 3.5 when creating the project from the template.

make sure .NET Framework 3.5 is selected at the top

I want the XNA template, so I click that, fillin the name, location and OK.

Horray, now I have a solution template filled in. now I can go back to step 3 of http://www.grumpydev.com/2008/12/26/surface-sdk-on-vista-x64/

I had to adapt these directions...

right click the solution and select “Configuration Manager...”
set “Active Solution configuration” to Debug

from “Active solution platform” select “New...”
select “x86” and OK.

make sure that “Platform” in the list below is set to x86.
set “Active Solution configuration” to Release
select x86.
make sure that “Platform” in the list below is set to x86.
close.

Now, I wonder if we can get this to run in the surface simulator?
http://msdn.microsoft.com/en-us/library/ee804880(v=surface.10).aspx

run the surfacesimulator first
then hit debug in the project.

Ok, that seemed to launch the blank XNA template in the simulator... so far, so good.

Let’s see if something works... found this quick start:
http://www.cs.uml.edu/~holly/teaching/91550/fall2009/handouts/lab2/XNA%20Quick%20Start.pdf

changing the sprite to “Resources\icon.png” (which already existed in the template) worked... I get the icon in the middle of the display.

WOOT!

Ok, next up, time to dig into XNA and see about graphics, and shaders, etc. I may start with simple particle sims.

patch straight from github!

2011-01-05T11:09:00.004-05:00

Wow, this is a cool trick...

1) go to any code on github
2) click on the "commit" link for any changeset
3) add ".diff" to the url
4) now copy the url and voila:

curl [url] | patch -d [dir]

e.g.

curl http://github.com/rails/rails/commit/268c9040d5c3c7ed30f3923eee71a78eeece8a8a.diff | sudo patch -d /Library/Ruby/Gems/1.8/gems/rails-2.3.5/lib/rails

So easy!

(courtesy of http://www.mattvsworld.com/blog/2010/03/version_requirements-deprecated-warning-in-rails/ and http://www.kartar.net/2009/09/output-github-commits-as-unified-diffs/)

how do you "freeze" individual gems in Rails 3?

2010-12-23T16:02:00.006-05:00

Ok, this wasn't documented anywhere, (the best I could find was the Rails 3.0 announcement and the Bundler rationale) so after reading the Rails guide I hacked around with it on my own.

In the old days (Rails 2) you could do something like this:

rake gems:unpack

and it would take the required gems and copy them into vendor/gems.

But Rails 3.0 gets rid of this handy feature in favor of Bundler's approach. Bundler's basic approach assumes that you'll always run bundle install in every deployment environment and that it can go to the internet to get the gems. Neither of these assumptions are valid in my case.

There is one way of doing this:

bundle install --deployment

but that copies everything, rails, ruby, 10,000 gems I've never heard of (well ya, I've heard of 'em, but sheesh) and nicely bloats the target environment. So it seems it is all or nothing.

Anyone else figure this out?

the single most important unstated rule about BDD...

2010-12-16T07:51:00.016-05:00

I've been co-leading a book club at work studying the RSpec Book. One thing that was difficult for me and other people new to Behavior-Driven Development (BDD) is that it often seems like the authors modify code in the BDD part of the cycle instead of waiting until the Test-Driven Development (TDD) part of the cycle.

For example, the first chapters dealing with BDD seem to make a lot of changes in actual code files and only at the end of Chapter 4 do they drop the inscrutable phrase, "now we have our first logical error." I know they mean "our first error that requires going into TDD" (because that's the next chapter), but how does this differ from all the code mucking they've already done? At first glance it looks like you can just skip around writing whatever code you want.

But I've found that the authors of the RSpec Book follow a really important rule that they don't come out and say:

When in the BDD cycle you can define new classes or methods (and even arguments to methods) BUT you are not allowed under any circumstances to change or define the implementation of those classes and methods.

This is super important because it explains why the authors sometimes write code while they're still in the BDD phase-- they aren't skipping around after all!

It makes sense when you think about it because BDD is all about the behavior and interaction between classes (outside), whereas TDD is all about the functionality and implementation of classes (inside). So, of course, in BDD you'd want the flexibility to define an interface on the outside as much as possible before being driven down into TDD.

Many programmers are confused at first about the difference between BDD and TDD and as a consequence, many more have blogged about the philosophical differences (google it!) but I haven't seen any clear rules before. This rule really helps solidify the differences in practical terms so you know exactly when you are in BDD and when you aren't.

are entropy and risk related?

2010-11-22T07:47:00.006-05:00

A friend of mine who is a plasma physicist posted a personal version of his presentation on the nature of the instability generated by a Maxwell's Demon wire-array. Well who doesn't like Maxwell's Demon, the little imp who goes around subverting Newton's Second Law of Thermodynamics by decreasing entropy?

Well since my friend's version had a lot more physics humor in it, I said he had invented a new field: "stand-up physicist" (although some may claim Feynman has first dibs on that)... my friend went on to say that next time he should relate the demon size collapse to our financial cycles. Heh heh...

But wait... I think that implies a really interesting question: is the concept of risk (including financial risk) somehow related to entropy?

In a time when our markets are being determined more and more by the math of physics and information theory, the idea that lowering financial risk is somehow akin to lowering entropy would be a very deep insight into the limits of a financial system.

Think about it. So far, Maxwell's demon hasn't beaten the 2nd law on the large... you may lower local entropy, but in the large, things always bounce back to a net entropy increase. Sound familiar? Markets and quants may be able to locally lower risk through use of financial derivatives, but ultimately, in the large, the markets always bounce back.

Wow. I just googled for "risk entropy" and apparently people in the field are already well aware of the connection. Well, even if it's not original, it's still a fascinating relationship.

Actually, this source summed it up great (duh!):

Any project, large or small is associated with expected and unexpected problems. The analogy mentioned above could be derived from the Second Law of Thermodynamics. The Second law of thermodynamics deals with a concept : Entropy. Entropy, in short, is the amount of disorderliness of the system. Entropy is also a measure on the information contained in an system. In information technology, entropy is considered as the amount of uncertainty in an given system. This has a defined relation, "As the amount of information increases, the disorderliness of a system (entropy) decreases".

compiling opengl-redbook examples on Lucid guest

2010-08-26T23:40:00.012-04:00

[UPDATE]

Rats! Turns out that any video setting or reboot gets confused with the nvidia driver present -- even though it isn't set, it doesn't play nice with the virtualbox driver... convinces it there is no 3D present and runs it in software (dirt slow). I eventually uninstalled it... maybe it's freeglut, but not sure. Needs more research. Later.

------------

I'm starting a class this fall in computer graphics and I thought I'd experiment with trying to compile some of the examples in the redbook. I have to wait for our class copies of VisualStudio (our course is taught targeting Windows), but I wanted to try some things, so I decided to use my Lucid Lynx Ubuntu 10.04 instance to compile examples.

First off, VirtualBox rocks. You have to hand it to the team because their 3D acceleration is strictly super-awesome... it lets me run full compiz settings while running on a i7 920 GTX 260 equipped Windows 7 host. It's fast! Like pretty close to native fast. So fast that I simply use ubuntu in VM mode instead of dual-booting. (Actually, I pretty much like everything about Windows 7 too, except maybe the filesystem changes, but there are more tools available for Linux, so it helps to have the best of both worlds.) Make sure 3D acceleration is enabled in your vm before you start; I also used the max setting of 128 MB VRAM. Anyway, back to the story...

Once in the guest OS, I downloaded the samples via apt-get, installed the usual libs... in this case the ones suggested here. Like the poster I tried the shipped makefile at first and got:

$ make
make: *** No rule to make target `$@.o', needed by `hello'. Stop.

Ug. I suspect some dialect of gnu make doesn't like that variable name or syntax, but gave up trying to understand it too much (yeah, lazy) and switched to the suggested longhand:

$ g++ hello.c -lGL -lGLU -lglut -o hello

Success! Then I ran it:

$ ./hello
OpenGL Warning: XGetVisualInfo returned 0 visuals for 0x24c10a0
Segmentation fault

Ug. Not what I wanted. Did some poking around and found that "nVidia and ATI "driver" installs on linux *replace* -lGL and -lGLU default Mesa installations with their own." So on a lark I tried

$ sudo apt-get install nvidia-glx-185-dev

And it worked like a charm!!

That's kind of obvious (because the VirtualBox 3D is supposed to be as clean a passthru as possible) but also kind of amazing (because the ubuntu guest is setup to use the virtualbox driver, not the nvidia 185 driver for ubuntu) -- it just worked... at least for this sample! :)

Enjoy!

DRM should live in the document, not the device

2010-08-07T14:31:00.002-04:00

I started taking a closer look at Kindle. A new semester is starting up and I noticed that several of the optional textbooks for the course are available on Amazon in Kindle format, so I was interested in seeing what the format could do.

Here's what I found:

If you are writing academic papers you can't cite Kindle versions very well. My suggestion to the Kindle development team is to expose "Locations" as standard URIs instead of the current proprietary bookmarks that only work on Kindle. URIs would be sharable and documentable and thus fit better with existing citation standards for electronic sources.
If you are using Kindle for PC, and you have a programming book with source code samples, you can't copy/paste the samples from the app to your editor. It's very ironic that you have to retype in code snippets when you're reading an ebook on your pc.
If you have a book and a friend asks you about it, a very common social case is to say "oh, here, you can borrow it, I'm not reading it right now" -- you can't do this with the Kindle because the DRM is in the device, not the document. I have to loan my entire library (and the Kindle too) in order to satisfy this use case. For this reason, I'm beginning to think that DRM (document rights management) should live in the doc, and merely enforce uniqueness and have nothing to do with "rights" or licenses per se.

I know the suits will be shocked at this idea: "but but, we were going to get all this money from individual sales forced on the customers!" -- no, I don't think you would. I think people will simply shrug and say "sorry, can't loan you the book, but you can buy your own" and then people will either say "ok", or "never mind".

However as books move to this new medium, I realize that the concept of a "library" simply doesn't work with DRM in the device. You can't loan licenses in the current model. Of course, the publishers again want to water at the trough of infinite profits -- but I think the reality will be far worse -- libraries will simply dry up and there will be little in electronic initiative to replace them.

Open formats may fill a little of the gap, but right now this is limited to academic papers and a handful of independent authors... it's hardly enough to keep libraries working... furthermore, open formats don't really encourage a library (except maybe digitally speaking) because you can just make copies of them.

If DRM lived in the doc, I could buy it, I could share it, but I couldn't copy it. If my friend had it, I wouldn't have it. This satisfies the "scarcity" requirement of the publishing industry (without scarcity, there is no value for books or any media).

But somewhere along the way, DRM became about "licenses" tied to devices. So I can't share, I can't own, but everyone can buy their own copies. Of course, the consumer market says "hey, well in that case, I don't use that book all the time, it's not worth the same price" -- and publishers are again shocked that consumers don't want to pay for ebooks at only 20% off.

Even the OReilly subscription model is hard to stomach at $300-400/yr for access to their entire library. I've paid a huge amount, but at the end of the year, I own nothing. It's really hard to see any value in that arrangement unless I'm constantly using 10 books every day, and even then... I can't share them with any colleagues. Then again, at $50-$100 per computer graphics book and factoring that such technology books are obsolete within a couple years on average -- after a certain point, subscriptions do look "cheaper"... but it's still pretty expensive from my perspective.

I ended up buying physical copies.

how to call lambdas without call()

2010-07-28T16:10:00.002-04:00

One thing that is rather nifty in javascript is the ability to assign anonymous functions to variables and then simply call them. For example, Protovis has this nifty method for creating a mapping function from the specified domain to the specified range:

var y = pv.Scale.linear(0, 100).range(0, 640);
y(100);  // 640
y(0);    // 0
y(50);   // 320

Neat!! Well, how about Ruby?

Well, I've been reading Metaprogramming Ruby (which is a really fun book so far) and we have lambdas. However, with a lambda, you usually have assign the lambda and then call() the lambda.

It looks like this:

f = lambda {|x| x}
f.call(10)  # 10

But I wanted this to be more like javascript's syntax, so I did some tinkering. Here's what I came up with:

the privacy of feedly feeds

2010-05-19T11:23:00.007-04:00

A while ago I confronted Feedly about an apparent hole in their firefox plugin on Twitter:

They claimed they don't store credentials per-se and after further investigation I believe them, but there's still something not quite right.

See, if you install the plugin, everything appears normal:

But when you turn on Firefox's "Private Browsing" mode and click the Feedly button, you still see your feeds!

Fortunately, after a while, Feedly attempts to update your feed and displays the login screen:

So this tells me that what feedly says is probably true, they don't cache your credentials in the plugin. However, they still apparently cache content from your feeds for a little while until the next refresh period. By itself, this content cache isn't a bad thing (it's a performance optimization and saves network bandwidth) -- but the fact that their local content cache doesn't respect privacy modes in the browser is somewhat disturbing... does that mean that they cache outside the browser's model? or does that mean that firefox doesn't secure local data? Either conclusion would be troubling.

Does this actually expose private information in practice? I can't guess how you'd exploit it, but it certainly doesn't give me a warm fuzzy feeling either.

SEO's "killer app"

2010-05-17T14:54:00.013-04:00

I was stymied when I saw this article about the recent connection between pesticides and ADHD. It wasn't the subject that I found interesting, but the hyperlinks... in particular as I was reading the article something stood out as bizarre:

"Chemical influences like pesticides used to protect produce from insects are believed to contribute heavily to this upward trend. Some scientists believe it may have an even greater impact than other environmental factors like video games, television and online personal loan advertisements that may have been linked previously to ADHD behavior." [emphasis mine and original link to http://personalmoneystore.com removed.]

Really? Online personal loan advertisements have been linked to inducing ADHD behavior? Of course, I was curious and clicked the link... but it only went back to a rather obnoxious personal loan advertisement on the host site. I looked at other links in the story and realized they all pointed to the ad in interesting ways. Like the text about US and Canadian children... "Canadian" links to "ontario-payday-loans" on the site. Oh... oh that's dirty.

I traced the whois for the site to adworkz.com, an online marketing company specializing in SEO (or Search Engine Optimization). My first thought was that they had simply scraped the original article off the Net and then inserted links -- but I couldn't find any exact matches for this story. However, another story on their site yielded a bunch of exact matches on other sites.

So maybe they are doing something especially innovative for SEO... they are paraphrasing articles, which seem to be legit news stories (I mean they are unless they are scraped) - and then overlay the article with links to their ads. Hey, I give them some credit, the links weren't completely random.

However, this is still a relatively "hard" way to get the numbers required for SEO campaigns -- rewriting pieces and hand-linking them isn't the most scalable business model. And because it takes "human-time" to do, Google can conceivably keep up with the Joneses. But what if there was an automatic way to paraphrase?

SEO's Killer App

Unfortunately, automated paraphrasing is exactly what is around the corner. Anne Eisenberg's article "’Get Me Rewrite!’ ’Hold On, I’ll Pass You to the Computer,’" describes how researchers at MIT and Cornell have come up with probabilistic methods of generating paraphrased content automatically. The code has been out there for a long time already and there are several interesting projects along similar lines such as Sujit Pal's Lucene-summarizer, Classifier4J, Open Text Summarizer and MEAD.

Imagine you are an SEO marketer... you suddenly have the ability to automatically paraphrase content from other sources (even the New York Times!), but because there are no identical phrases, Google can't detect (and therefore can't block) your ranking. In fact, it's difficult for humans to tell which is the paraphrase and which is the original... and unlike most scraped content feeds, it looks completely legit to readers who stumble across it via search engines.

Mix this with other SEO techniques, such as dynamic domain generation, and sell into several different TLDs and search engines would not be able to detect or defeat the flood of seemingly legitimate links.

So why am I helping the "evil" SEO marketers out there by foretelling this unstoppable weakness and possibly bringing about the demise* of search engines?

*Yes, this is what unrestrained SEO might do. Search engines are only used by people when they yield useful results -- when every link goes to an ad, a trick, a deception, people will stop using the service. Notice the chilling effect that telemarketing has had on land-line phones (many people have cell-phones only now), or door-to-door salemen had on a previous era).

The Counter Move

Fortunately, search engines have some awesome counter moves at their disposal. It's the same thing that identifies spam email so readily: the target url. SEO marketers have many many tricks, but the one thing they can't disguise is the link they want you to go to. So this is how you identify them:

"SURBLs are lists of web sites that have appeared in unsolicited messages. Unlike most lists, SURBLs are not lists of message senders... Web sites seen in unsolicited messages tend to be more stable than the rapidly changing botnet IP addresses used to send the vast majority of them... Many applications support SURBLs, including SpamAssassin and filters for most major MTAs including sendmail, postfix, qmail, exim, Exchange, qpsmtpd and others."

consumers hooked on oil? please.

2010-05-12T23:39:00.004-04:00

I don't agree with Peter Maass' assessment that

"because when you kind of get down to it, American consumers do want to have their gasoline."

As a consumer, I'd love to spend less on gas. One practical example of how I could do this is to telecommute to work (not every job can do this, but some can), but most corporations don't believe this is an option. How about WebEx meetings instead of business travel? Many don't believe this is as good as face time and are willing to pay for air travel. There are many other alternatives I could list that either require unacceptable tradeoffs or drastically reduced standards of living or drastically higher expenses to support alternative energy. The infrastructure for what he wants simply doesn't exist yet.

So what are consumers going to do? Of course they're going to want their gasoline because there is no way to make a living without it. Give us an alternative before accusing us of inaction! Personally, I think rising gas prices will ultimately motivate alternative energy. As alternative energy becomes cheaper than oil, consumers will be happy to switch, as will the corporations and markets that connect them.

But Maass' premise is insulting. He thinks we can afford it and we're just being petulant. Instead, he should be focusing his energy on the existing economic system that was built on oil. Of course we want to get off oil dependency as soon as possible and move towards sustainable energy, but Maass wants the consumer to single-handedly bear the cost of switching without any help from the structure of corporations or governments.

Most of us can't afford that and Maass is forgetting where the consumer's power to spend money comes from in the first place: the ability to earn money and spend responsibly within that structure, not apart from it.

Ben Ward says "Link to it!!"

2010-05-04T11:23:00.031-04:00

Ben Ward's post about the difference between web-like rich applications and real web content is awesome.

I think it pins down some distinctions that have been brewing for a while, and solidifies several of my own thoughts about it.

For example, his statement:

"If you want to build the most amazing user interface, you will need to use native platforms."

This is the fuel behind java applets, Flash, SVG, VRML and every other extension to the web developed so far. It's about presentation. It's something the W3C does exceptionally poorly, and I think they shouldn't attempt to do it at all. It's also something that every tools company has struggled with: the only way to extend kick-ass native platforms into the browser is to create a new plugin. The only way to get a new plugin recognized is ubiquity among the user-base. Flash arguably won that battle, but it's a losing war in the long run because now it's Flash or nothing. The other "open" alternatives are attempts that are valid in their own right, but have never been quite ubiquitous enough. And HTML5 simply shuffles a bunch of capabilities from plugins to the browser, which certainly makes those capabilities ubiquitous, but doesn't necessarily make them better, or even more open (as Ben points out about H.264).

But Ben also makes a brilliant case for content:

"Want to know if your ‘HTML application’ is part of the web? Link me into it. Not just link me to it; link me into it. Not just to the black-box frontpage. Link me to a piece of content. Show me that it can be crawled, show me that we can draw strands of silk between the resources presented in your app. That is the web: The beautiful interconnection of navigable content. If your website locks content away in a container, outside the reach of hyperlinks, you’re not building any kind of ‘web’ app. You’re doing something else."

This idea, the linking of content, is exactly what the W3C has always excelled at. It's awesome, the idea that I can simply link and connect pieces of content from all over the world and make my own connections and conclusions and in turn become a piece of content for someone else. This was Tim-Berners Lee's original vision of the web and I agree with Ben, it shouldn't be lost. We should fight for it. Maybe the real problem is that presentation and content have gotten so tangled and confused that we've forgotten the difference?

This galvanizes me to talk about some experiments I've been doing recently to make simple websites that make the distinction VERY VERY clear. Let me show you an example:

This semester, I had to produce "mini-sites" for class homework assignments. These mini sites had to basically run as static html because they had to be viewed from the local filesystem of a TF after being zipped up and submitted. They had to be viewable on the wide range of platforms used by the TFs, which effectively meant vanilla HTML/CSS, but I didn't want to have to code a TOC by hand either. What I came up with was an xml file tuned exactly to my content.

Take a look at the finished html page: writeup.html

This was generated from a handcrafted xml file: homework.xml. (You'll have to "View Source" to see the xml if you're in a XSLT capable browser like FireFox because it will automatically render the HTML for you). There are very few presentation aspects in this xml file, and quite a few "new" custom elements I created that are specific to the structure of my assignments. It's small and ad-hoc, which is fine. It's basically a micro-format without all of the extraneous divs.

I render it with xslt using this file for the presentation layer: homwork.xsi. This file contains all the rejiggering and mucking about that presentation layers require. Even so, I tried to make it as clean as possible by utilizing CSS where I could and getting creative with XSLT. It can even provide microformats-compatible semantic markup in the generated div elements -- the best of both worlds! Finally, I generated the static html file using a very simple ruby script: genhtml.rb.

Sure it's rough and simple (I didn't have a lot of time), but it highlights what I want to show: we don't need fancy web platforms or standards to separate content from presentation. We can do it right now, today, with the tools we have, it just requires the right mind-set.

I'd love to see WebGL, Processing and Flash sites that have separate RESTful urls that flay out all their internal content in browseable xml or JSON microformats, easy to link to, easy to transform and include in other pages or references. (btw, Processing already does this somewhat by putting links to the source code in the applet page -- brilliant!)

I'd also love to see brilliant new presentation layer technologies form, like WebGL, Processing for the web, javascript, Flash, webkit, etc. because I want to innovate and create that new killer interface and interaction model for my content that no one has thought of before and no one else has implemented before!

I think Ben's post and my experiments show a way that we can have both of these things at the same time as long as we don't get confused and tangled up in the standards war or the RIA platform war and forget the difference between content and presentation.

OReily's Internet Operating System FTW!!

2010-05-03T21:01:00.003-04:00

I'm glad OReily wrote his post "The State of the Internet Operating System." I agree with it a lot.

I said this years ago. In fact, I'll repeat the idea here: just like early video devs went through hell writing to non-standard graphic card memory models, early web devs have had to struggle with non-standard serialization and presentation formats.

And just like back then, a crop of "device drivers" slowly formed (first from DOS systems like Miles Sound System, etc. then later via the OS) -- well it's happening now: people fed up with the conflicting standards of the w3c and webdev-hell have had it and are starting to create "web-standard" independent "drivers" -- like webkit, like jquery -- that don't care what version of HTML browser or CSS you really have, that degrade gracefully by offering "hardware abstractions" (or in this case browser and w3c standards abstractions!) And javascript is rapidly becoming the language of the new platform because no-one else is moving fast enough.

In my mind it's time to put the W3C to bed where they belong -- leave them as keepers of a data standard to the semantic web, but for crying out loud, give the presentation devs a standard platform we can actually code to with the solidity and flexibility of postscript or renderman!

IMHO, the next steps (great leaps forward) are standard serialization formats. Who gives a flying &#&#@ if you are doing a GET or a POST or a JSON or an xml, or java-to-javascript, or what the hell have you -- SERIOUSLY, it's as insane as memory management was before contiguous memory controllers (does anyone remember those days?) -- it's all state serialization, it should have a freakin standard way of writing and reading no matter where it comes from or where it goes to -- I'm not talking implementation details, I'm talking raw syntax.

ActiveRecord and JSON begins to get close to this, but we've got a LONG way to go before I can just say "give me an address from the database and post it to that RESTful service and write it to a memory cache over there -- and it's all the same damn syntax! Why reinvent the wheel 80 bazillion times?

/rant off

risk management implies socialism not capitalism...

2010-01-08T07:02:00.004-05:00

I've been hearing several stories on NPR recently that question the ideas of an unregulated free-market economy given the recent financial meltdown. The argument is that laissez-faire markets have been shown "not to work," there are now new doubts about whether markets are the most efficient system for managing resources.

Ok, so why is this disturbing? Because I think a critical distinction is being lost in the debate between free-markets and controlled economies. The critical distinction is a concept of risk.

Unregulated free markets work. They work well. They are extremely efficient. Nothing I've seen disproves that. However, they can be extraordinarily brutal -- usually when speculative bubbles pop. The free-market theory has been that less regulation will allow these bubbles to pop sooner and smaller -- but as recent experience has shown, that's not necessarily the case. A free-market's ideal is in information that represents actual value, but it's entirely possible for prolonged periods of speculation to exist and even grow, fueling such bubbles. When large bubbles pop, huge sections of the economy may be at risk. If this were allowed to happen, the results would be catastrophic, but they would ensure a quick demise to everyone who touched (and fueled) such speculation. This doesn't mean the concept of free-market is broken, or it's not efficient. It is cruelly efficient, but what we are really talking about is risk, not the free-market!

In fact, let me make a bold thesis: all forms of risk management move away from free-market capitalism towards socialism.

I'm not necessarily against socialism, but let's look closely at this new distinction that is absent from the current dialogue: if you embrace risk management, it means ultimately that you are diffusing risk to other areas, you are in effect damping all the wild fluctuations of the market into a smooth, level field. Hedge funds did this by intricately linking and amplifying thousands of previously unrelated investments. The bailout did this by spreading the losses over all American taxpayers. The extreme end of this arrow is zero-risk: zero development, zero capital, zero life. The extreme end of the free-market arrow is maximum-risk, which could be heaven or hell, or likely both over time.

The math of hedge funds and derivatives is fundamentally unstable - in every place it has been used, the results have been unpredictable and disastrous from the beginning of the very first hedge fund to use "zero-risk" maths, which lost billions of dollars in a few months. I believe this is because people are missing the connection between risk-management and socialist-economics. One implies the other automatically. This becomes painfully obvious if you look at the basic premise of hedge funds: to create zero-risk investment instruments... how is such a thing possible in a capitalist system which is built on risk? It would be like trying to build a casino where there is zero-risk... would you call it gambling?

Realistically, we will always have a mixed economy in some respects. The question is not whether one socio-economic system is better than another, it's how much do you want of one or the other. Or put simply, how much risk can we afford? I don't think zero-risk is necessarily a good place to aim, because we've seen exactly how that turns out.

Avatar is Awesome!

2009-12-30T11:18:00.002-05:00

With all the awesome effects I was worried that the story might have floated off like one of the mountains in the movie -- however Cameron knocks it out of the park by providing a solid and emotionally compelling story that touches on humanity and forgiveness over greed and destruction.

The technology allowing the actors to transfer their performance to their synthetic counterparts is amazing. In that respect, this movie reminded me of Jurassic Park -- on one level you're talking about genetically engineering dinosaurs -- but on another, they really created compelling animated dinosaurs!

Well, in the same way, Avatar shows people transferring consciousness to synthetic alien/human hybrids, but on another level, Cameron really developed the technology to transfer the actor's "consciousness" -- their performance to the CGI models. Like Jurassic Park, that's nothing short of amazing.

ontological paradox...

2009-12-01T14:57:00.003-05:00

Noticing today's RocketBoom on the Ontological Paradox made me start musing about one of the examples, specifically:

On his 30th birthday, a man who wishes to build a time machine is visited by a future version of himself. This future self explains to him that he should not worry about designing the time machine, as he has done it in the future. The man receives the schematics from his future self and starts building the time machine. Time passes until he finally completes the time machine. He then uses it to travel back in time to his 30th birthday, where he gives the schematics to his past self, closing the loop. Of course, the schematics must have come from somewhere.

There might be a simple way around the problem of where the information comes from if you accept the "many-worlds" interpretation of quantum mechanics:

Let M be the multiverse, or the set of all universes.
Let u be a universe in M where the man successfully develops the time machine on his own.
All he has to do is find any other universe u' in M such that the current time in u' is equal to a previous time in u (i.e. 30 years ago).
If, technically, he travels across universes from u to u' (as opposed to really traveling in time in his own universe) then the information (the plans) really come from u, not u', so there is no spontaneous generation of information.
The fact that the plans generate a "time" machine which allows the loop to "close" is completely accidental since "time" travel would in reality continually travel to another universe (e.g. once in u', he would need to find a u'', etc.)
Also, because he is traveling between different universes (as opposed to traveling backwards in time) he doesn't break any known laws of physics.

There are some problems with this way of thinking about things though...

If there are universes "in the past" and there are no limits to how far we can go "back", it implies that there is always at least one universe right now that is undergoing the very first instant of the big bang. I don't think "many worlds" allows such a juxtaposition. It's probably not a given that a universe u' can always be found.

You might say that this doesn't matter, because he's only going back 30 years... however, once the loop is "closed", it might easily execute more than n = (the age of the universe/30) times... if you imagine it takes n universes (u₁...u_n) to perpetuate this cycle, then the last universe u_n in the chain must have been undergoing the first instant of the big-bang when he left his original universe u.

"why do we seem to only know about one universe"

There has always been another weird problem with the multiverse interpretation... supposedly, different event probabilities create different universes in which each probability actually happened. If that's true, then why do we seem to only know about one universe -- i.e. why does our consciousness seem always to travel with this particular branching path and not one of the others?

Then I realized this isn't such a problem after all: we might only remember this universe because at any given moment our memories are the sum total of our experience along only the path we are in. Our other "selves" would likely have different memories, but all similarly self-constrained to their paths.

There's no way to "peek around the quantum veil" as it were.

journalism in crisis...

2009-09-20T17:56:00.012-04:00

A while ago I had heard Bill Moyers speech regarding "Journalism in Profound Crisis". I knew the dangers, but today the full weight of his speech hit me while reading a very tame and completely unrelated "news" article linked from a friend's facebook account.

The story, which appeared on the headline Business section of MSNBC.com on Sept 20th, outlines how a family recovered from $100,000 debt in 5 years to be debt free and happier and more responsible than they had been. At first, I thought it was just an uplifting story... but then I started to notice some odd things about the story: for one, it was copyright by CreditCards.com. Second, the story disclosed that the family had won a PACE award which was judged by one of CreditCards.com's senior reporters.

Since when do private credit card companies hire reporters?

"How many other news stories are really embedded corporate press releases?"

The story explains that the family was holding even until "credit card companies [began] raising their interest rates" even though the family had been current on all their bills. Credit card companies proactively raising their rates has been described elsewhere as a response to the Card Accountability and Disclosure Act (CARD) and the state of the economy in general.

CreditCards.com identifies itself as a broker between consumers and credit card companies. So it has a vested interest in protecting the interests of credit card companies who are it's customers.

While other stories have focused on the unfairness of raising rates for consumers who have always paid their bills on time, the CreditCard.com story quietly bypasses whether the rate increases were fair or just, and just commends the family for paying their debts without raising a fuss.

This is brilliant marketing, but is it news?

What's shocking isn't the story they wrote, or that they are one of MSNBC's advertising sponsors, it is that their "story" appeared as headline business news on MSNBC.com, a company that touts itself as "one of the most honored news organizations." Ordinarily this kind of PR story appears (or is credited) exactly where it belongs, on PR Newswire. The fact that it was published as a story subject to the same journalistic integrity that MSNBC.com is so proud of is deeply disturbing.

How many other news stories are really embedded corporate press releases?

Bill Moyers wasn't kidding about the threat to journalism. The internet gives us a lot of power to write and read stories almost as they happen. But it also gives corporations the power to pretend to be journalists without fully disclosing their motives.

a question about the "holographic principle"

2009-08-28T12:00:00.008-04:00

I've been reading Leonard Susskind's book "The Black Hole War" and trying to wrap my head around this mind-bending concept he presents called the Holographic Principle.

Roughly, the idea is built from the properties of entropy and temperature surrounding black hole physics. There is a curious result that a black hole's entropy is proportional to the area of its event horizon rather than the enclosed volume. Susskind goes on to explain that this is effectively as though all the possible information about the inside of the black hole were written on it's surface in Planck area sized quanta. So this is a little surprising, but you expect that with black holes. What's really weird is that this ends up letting you describe any volume of space with a 2D encoding of the surface. Is the 3D universe just a projection of this 2D encoded surface? Weird!

Ok, not so weird... I mean, I'm a programmer, I'm very comfortable with transformations between 2D and 3D. Most of the ones we use in computer graphics are called "information losing" transformations, because once we've rendered a scene, you can't "walk around" it from the back or sides. However there are "information preserving" transforms that tell you how to encode exactly the same amount of information in a lower dimension, so this isn't too strange for me to grasp.

Another paradox?

All this has physicists wondering if the underlying nature of reality is really a 2D surface off at the edge of the universe. While I could see that transform springing out of the math involved (not that I've done any of it, mind you), I have a problem with literally believing this is the structure of things.

What is hard for me to grasp is that a literal interpretation would seem to imply that information can propagate faster than light in some cases, which is another paradox! Think of the following thought experiment: say I'm playing billiards with Susskind and I sink 5 balls in one shot. Apart from having to be an amazing billiard player (I'm not), all these balls would have physical interactions. According to Susskind, these interactions are really happening at the bounding surface of the universe in a sophisticated 2D "holographic" encoding and we just might think it was happening in 3D here. Fine.

But wait, how do you know that the interactions are all happening locally on that distant surface? If one billiard ball is represented by bits on one part of the sphere and another billiard ball is represented by bits on the opposite side of the sphere... how do they interact in a short amount of time? You might be talking about interaction effects on the surface that might be vast numbers of parsecs apart.

The interactions in our notion of space are also very fast... fractions of a sec. How can interactions happen so quickly if the underlying "real" medium was so extended? Does the underlying information travel faster than light when it interacts? (i.e. "Spooky" action at a distance?)

This is potential paradox. The only ways out seem to be 1) saying that the speed of light doesn't apply to propagation (interactions) in the underlying 2D surface, or 2) all information is encoded such that it's spread is somehow constrained locally with respect to the surface.

This tends to make me think that the Holographic Principle is a mathematical relationship more than representing any underlying nature of reality. But then again I'm basing that off of naive assumptions of how optical holograms work and simple 2D and 3D projections -- perhaps there is nothing so simple about the nature of the projections Susskind is talking about?

What do physicists think of this? Is it a real problem?

Is health care "infrastructure"?

2009-08-13T08:18:00.050-04:00

One of the more hotly contested issues in the current debate over the health care reform is the amount of government control over health care. Obama's plan has been labeled Marxist by his opponents, implying that it is nothing more than an attempt to nationalize health care, a term which conjures images of evil Socialist countries greedily dividing up someone else's profits.

But the government occasionally has legitimate reasons to provide common infrastructure. One example is the US Postal Service. Ben Franklin saw a good communications infrastructure as vital to the growth of the nation, and he was arguably correct. Most people don't think of the US Postal Service as being a "nationalized" institution because there are still ways to compete (FedEx or Mail Boxes Etc.). Competing with a government-run option is entirely possible.

Who said anything about "government run"?

Obama proposed the "public option" as a government-run non-profit option that will encourage competition with private insurers. He didn't say anything about the government taking over all of the health-care industry (a truly Marxist "nationalization"), but the conservative Right has interpreted his plan to mean exactly that.

The Right isn't completely "wrong" either -- it is a reasonable guess based on the history of competition in the insurance industry, which tends to take the form of acquisitions and mergers, leading to a few large fish and no small fish. It is ultimately a numbers game: the biggest numbers win, plain and simple.

Insurance from the individual's viewpoint is simply a way to pay a "guaranteed and known small loss to prevent [the possibility of] a large and possibly devastating loss." The individual doesn't really know when or if something bad will happen to them specifically, but it always pays to be prepared.

From the insurer's viewpoint, this looks a little different. If I were a brand new insurance company and I had only one customer, offering him insurance at competitive rates would be a huge risk on my part. There's only one way I can make this work, and that is to get lots and lots of customers.

Now, lets say I have a lot of customers, but I'm a publicly held company. My shareholders expect profitable quarters. I have only a few ways of making a profit:

increase the number of low-risk customers
increase premiums
decrease coverage (or increase coverage exceptions/pre-existing conditions)
put a greater portion of holdings into higher-risk investments (higher risk!)

By removing profit as a motive, Obama's "public option" would not face many of these pressures, so presumably it could provide more service for less money. The downside is that the private for-profit health care industry would be hard pressed to compete without becoming non-profit themselves -- so it is likely that the health care system becomes nationalized gradually over time as people move over to the "public option" even though that wasn't the original intent.

Is this a bad thing?

Getting back to the idea of infrastructure, this is where I'm torn. I hear things like Palin's quip about "death panels" lamenting the idea that the government may be deciding who is fit to live or die -- she raises a completely valid concern but does not mention that private insurers already have such panels and that they are answerable to no one, least of all the customers they serve. The fact is that inevitably any organization involved in managing insurance has to make life and death decisions about providing care coverage. Where is the outrage against existing "death panels" in private companies like those exposed by Michael Moore's documentary Sicko?

"...inevitably any organization involved in managing insurance has to make life and death decisions about providing care coverage."

One major difference between private companies and a national health care system is that the private company gives you no recourse, no voting power, no voice and no choice*. A national health care infrastructure might at least be influenced by voters instead of shareholders. Assuming, of course, you don't mind a little Socialism in your health care. But that's really scary to a lot of Americans...

*Capitalism says you do have a choice, you vote with your dollars -- however, there is a defacto cartel in the insurance industry because once you have a condition you are unable to shop for competitors due to the exclusion of "pre-existing conditions" -- hence, there is little free-market competition to begin with and none when you really need it.

Insurance is already a Socialist construct!

Think about it. Insurance basically takes a fixed amount from everyone who buys into it and tries to allocate it to where people have the most need. This is the essence of a Marxist dynamic and just like its political counterpart, is a reaction to the harsh unregulated realities of real life. Even the most ardent Capitalist has some kind of risk management in the form of insurance -- to not protect your investments is just plain stupid.

The reason that most people don't rebel against the idea of insurance is that (unlike Marxism) it is (or was) completely optional. You don't want to pay, fine... take your chances, or manage your risk some other way. This sounds fine in theory, however, for most people that didn't have insurance, that simply meant taking a free ride on all the rest of us... hospitals still gave them care, those costs were still passed on to insurers and those of us who pay for insurance get the bill in the end.

The only way to make insurance rationally opt-in is to always deny care if you can't pay, even in an emergency... a stance that is so controversial that most health care organizations refuse to consider it.

---

In the face of all these things, I just don't know what the right answer is. I tend to think that health care could reasonably be considered infrastructure by the government which has a vested interest in keeping its population healthy just as it has an undeniable interest in national security.

It seems unbelievably cruel to say "if you can't pay, you die" -- yet that is the only way a truly capitalist system would work. And I think the insurance industry has far too much invested in the profits of the existing system to turn it over to the government without a fight.

Perhaps this means that the matter is moot, but I like to believe that this debate will at least get people thinking about the costs involved and what kind of system they really want.

The Dalai Lama in Boston!

2009-05-03T08:53:00.020-04:00

Yesterday I went to hear His Holiness The Dalai Lama speak at Gillette Stadium in Foxboro. He is one of the most down-to-earth spiritual leaders I have ever heard. There is no arrogance or appeals to incomprehensible mystical forces-- just plain simple concepts.

Before the Afternoon Session

I arrived for the afternoon session around 12pm during the lunch break. The crowd was pretty diverse... Tibetans, a few Buddhist monks, some yoga and tai chi people out on the grass, New Agers, hippies, yuppies, families... people. I wandered around the level behind the seating where all the food vendors were-- they had set up some Tibetan cultural displays and shops. (It was a little odd seeing these nestled among the pretzel vendor and the McDonalds.)

The event was sponsored by the Tibetan Association of Boston and funds raised are going towards the construction of a new Tibetan Cultural Center in Boston. I went out and found my seating and listened to a concert of Tibetan culture being performed on the stage. Here's a sampling:

The video clip shows some dancing and the Tibetan musician Penpa Tsering. Some of the flute music reminds me of Native American music I grew up with in Arizona. Indeed, another one of the musicians at the event Nawang Khechog collaborated with Navajo flutist R. Carlos Nakai. Nawang performed an song dedicated to the Dalai Lama.

Penpa Tsering's piece with the stringed instrument really fascinated me... it reminded me of an oud, both in the intonation and in the rythmic style. It got me thinking about the connections in a paper on Russian music history I'm writing -- Arab influences traveled up through the Crimea from Persia and along the Silk Road later inspired composers like Rimsky-Korsakov in music like Scheherazade -- did they also make their way into the musical traditions of Tibet? Certainly the historical trade routes suggest that.

Immersed in these thoughts, I realized that the concert portion was over and The Dalai Lama's talk was about to begin.

"The Path to Peace and Happiness"
(The following is my recollection of parts of his talk, so I may paraphrase or interpret or quote parts out of order -- this is not a transcript.)

He walked out onto the stage and sat in his chair. His movements were calm and unrushed, without any tightness of formality. He drew his legs up into a crossed position in the chair, wrapped in his robes and searched around for something in the cloth bag he brought with him... then he found it and let out a chuckle to himself and everyone joined in when they saw that he had put on a New England Patriots cap to keep the sun out of his eyes. He gave a big thumbs up and a smile.

He started very simple. "We are all human beings." He pointed out that if he said "religious" it would exclude all the "non-religious", or "Tibetan" might exclude all the non-Tibetan or anti-Tibetan... so our commonality first and foremost is human. We all want to be happy. Our happiness must not be at the expense of others, for they too want to be happy.

He said that some people have attributed magical healing powers to his blessings, but he explained "if that were so, I wouldn't have needed surgery," and chuckled. However he offered another interpretation -- he noted that his doctor remarked how young his organs looked relative to his age -- this perhaps can be attributed to long practice in calm thought and compassion. He noted that scientists have studied the effects of stress on people and the positive health effects of happiness, so this is not too hard to believe.

The Dalai Lama is different than most spiritual leaders in that he embraces and supports science. He enjoys connecting scientific and Buddhist insights, and it seems for him to be a way to more deeply understand the nature of things. This is very refreshing in a world where the choices seem increasingly between spirituality devoid of reason and reason devoid of spirituality. (This effort goes beyond such narrow confines as "religion" -- for example, he is helping to launch a spirtual center at MIT focusing on ethics in business and science.)

Tying into his focus on essential humanity, he notes that ethics, compassion.. these things are more basic than institution, religion, etc. We need them as human beings. In this way, the Dalai Lama is trying to broaden his message beyond Tibet, beyond Buddhism... beyond religion... to something that applies to all of us in the world.

Then he gave some advice for how to attain happiness. For one, if you can widen your focus from yourself to the larger context, this can reduce your stress. He gave some examples, but I thought of the most striking example I know, the famous picture "Earthrise". When you realize your troubles next to all the other thousands of human beings who are going through similar things... it helps to calm your stress.

He noted that sometimes scientists and politicians tend to use the words "I" and "me" a lot. These words narrow one's focus, making small problems seem much more stressful.

Another thought he offered was that even when there is very bad news, there are also positive effects. For example, although being exiled from his homeland was bad, it has given him the opportunity to talk to many different people and spread awareness in the world. So, every time he hears or experiences something bad, he tries immediately to shift his perspective to notice something good that will come out of it... this also calms stress.

He did touch on the question of "freeing Tibet"-- he said that Tibet would even be willing to be part of the PRC if they were allowed to keep their culture and their language. Of course, this opens up a much larger and more complex set of issues that he didn't get into. He did invite observers from all nations into Tibet -- "if you see that conditions are as the Chinese say, then my information was wrong and I humbly apologize." I assume he is speaking here of the spiral of violence between rebel and PRC forces.

Perhaps it is impossible at this point to separate The Dalai Lama's message from the conflict in Tibet, however he says he wants to keep the dialogue open, searching for a non-violent "win-win" solution that benefits both sides. There is much I don't know or understand in this conflict. The U.S. and China are interested in this area strategically for different ends... for China, it serves a similar strategic interest to our historical "Manifest Destiny," which places us in a difficult position to claim moral superiority or defense of an indigenous culture. Yet, we have a greater recognition and respect for these cultures now... to lose them would be a incredible loss to the world. I think many people agree that we should do all we can to preserve these traditions for historical reasons if nothing else.

But, at least in The Dalai Lama's case, these traditions are surprisingly modern in their application. Regardless of our technology, the principles of how we relate to each other and form meaningful relationships have not changed in thousands of years. The Dalai Lama is remarkably clear in applying Buddhist principles to everyday modern life.

We need his voice in this world, now more than ever.

web 3.0 needs a javascript package manager

2009-03-15T12:52:00.007-04:00

Javascript has gotten huge with Web 2.0 techniques. Libraries like jQuery, Scriptaculous, etc. etc. have helped redefine the role of Javascript from being a rag-tag script-kiddie language to becoming a viable platform for the web, competing with the likes of Flex and Silverlight. Javascript is getting serious attention from Google (ala V8) as a serious language with serious compiler design. In short, it's a glorious time to be using Javascript on the web.

So while the iron of Javascript infrastructure is hot, I have an idea for the next generation of the web:

the time has come for a javascript package manager.

No, I'm not talking about javascript loaders like jspkg or sprockets (although those are awesome efforts). I'm talking about a real package manager like Debian. Let me describe the growing problem and why I think a package manager might be a good solution.

The Problem:

Different sites require certain versions of scripts to run. Right now, site maintainers do this by downloading a copy of the library to serve from their site. This is fine if the site is simple and self-contained, but if you add several portlets to your site (a couple ad-rotators, some captchas, social site scripts), pretty soon you run the risk of accidentally including several copies of the same library in your page, all from different domains.

This sops up an enormous amount of bandwidth. Rich pages routinely download 1-2 Mb per page load... this is only going to get worse.

The Solution:

What we need is a package manager for Javascript like Debian's APT or Ruby's rubygems. Some way that the page can say "I need so-and-so version of this Javascript library in order to run." Then the browser can manage these javascripts in a secure cache -- if you have already retreived the version required, good. If not, then the browser will get a copy for you. The browser only gets new javascript if the page contains an updated version dependency. Furthermore, if a library depends on another library, the version dependency tree can be managed and sub-libraries can be automatically loaded once and only once.

Because the browser would be taking a more active role in managing scripts for separate domains in the same store, some work along the lines of Chrome needs to be done to make sure that script execution across multiple site clients remains secure and robust. Chrome is already leading the way on this, so I think it's doable.

A robust, integrated package management system for browsers would reduce bandwidth costs and at least double or triple the loading speed of AJAX web applications. That's something everyone would like.

Communism 2.0? More like 0.2 alpha.

2009-03-13T11:56:00.010-04:00

Bruce Sterling's post, "Communism 2.0" over at WIRED caught my eye today. He talks about a new resurgence of communist ideas thanks to the global market meltdown -- Marxist philosophers are meeting in London this week with the thesis "from Plato onwards, communism is the only political idea worthy of a philosopher."

When the Soviet Union "fell" down many people thought this meant that Communism utterly failed (according to our vastly simplified propaganda). Well, now that the global market meltdown is laying waste to the economy, many people are saying that Capitalism has utterly failed (according to their vastly simplified propaganda).

This is dangerous talk, and not simply because of "our side" vs. "their side"... it's dangerous because irrational debate of utopia/dystopia is still volatile enough to end millions of lives in this century just like it did in the last century. Capitalism is just as loaded a word as Communism thanks to the Cold War, but if you strip away all the rhetoric and attached meanings, there is a simple fact at the core of Capitalism that is testable and proven:

Capitalism has a means for generating income, not simply dividing it.

Communist theory is oddly silent on the subject of generating income. The idea is that everyone will simply do the right thing. Marx neglected to say anything specific about how his Communist ideal would be attained... the underlying mechanism is left for the reader to "feel" as would a 19th century Romantic. Marx says that in the perfect Communist society, resources are distributed exactly where they need to go and people do exactly the work they need to do since they work for each other.

But how? How does a resource distribution system just magically know how to work? The 20th century answer to this was that an "appointed" leader makes the decisions for you. However the whole period from the 18th to the 20th centuries was about showing the flaw and futility of top-down aristocratic rule. Even when the ruler is kind and "works for the people", it doesn't work. Things change faster than a central authority can react -- if you can't react, you don't adapt -- you die.

Some would argue that matches the evidence of what happened in the USSR and nearly happened in China before Deng Xiaoping's reforms. Or what is still happening in Cuba. There should be little doubt at this point that top-down communism (like any aristocracy) simply does not work. It's not because we didn't try hard enough, or weren't selfless enough.

So, is there another way that Communism might work without being top-down? Something closer to Marx's ideal, rather than some of the transition states he described? Well, think seriously about how that ideal would work... it's a difficult problem to solve.

The only underlying mechanism I can think of is a distributed network of technology and humans that can make their own decisions and adapt locally to changing situations. But this sounds an awful lot like a democracy... there's no one to "enforce" that you do things for someone you don't interact with. And if you follow that idea further, you ask how will resources be optimally distributed in such a network and you come straight back to something that sounds an awful lot like a market driven economy.

But the Communist philosophers haven't gotten unstuck from the 19th century to answer these difficult questions. They are still arguing that you should believe them because of their passions. Until they can describe an underlying mechanism of Communism they are simply promoting unfounded speculation as a socio-economic theory and playing a very dangerous and foolish game with our lives in the process.

This crisis is real. The millions who died in the name of 19th century utopian ideologies were real. It's time to "set aside our childish things" and face these problems rationally or embrace utter destruction as a species.

the bigger picture...

2009-02-23T17:15:00.003-05:00

The previous post about the importance of technical managers is just a small example of what I think is the importance of mathematics and science education in general.

At first people don't pay much conscious attention to design principles. They simply do whatever works. For a time this is sufficient. But soon people begin to think about efficiency. Maybe it's their competition or scarce resources, or their customers can't pay as much as they once did. Whatever the motivation, when people first start considering the constraints of a problem and how to solve those problems esthetically, designers are born.

But there is another layer beyond this. When economies drive down the costs of design and organizations must no longer consider the merits of a few designs, but rather must select against an endless sea of millions of designs and practices, the question becomes "what is the optimal design?"

This is the point where mathematics gets involved. Math is the language for describing systems and calculating their optima.

This pattern should not be surprising to anyone, but sometimes the conclusions are surprising. For one, it seems that any sufficiently advanced and refined problem domain comes to mathematics. Whether it's how to ship a million packages a day, or how the basic building blocks of life are sequenced in various creatures -- large scale almost begs for mathematical solutions. For another, it implies that no matter what your particular artistry entails, if you follow it long enough, you will end up having a mathematical interest of some sort.

This isn't that remarkable if you look at the trends of modern science: biologists are talking with computer scientists are talking with physicists... and all of these are talking with mathematicians. Even linguists, who started with a love and study of languages and etymology have branched into computational linguists who study language origins using statistical methods in mathematics. Some of the fields developed their own systems and specializations in the infancy of their fields that are only now being realized as special cases of more generalized fields in mathematics. Core topics like Set Theory and Ring Theory are now prerequisites for dozens of studies in the arts and sciences from biology to finance.

The lesson is clear: if you want to be the best at what you do, science and math education are the prerequisites.

non-technical managers...

2009-02-23T10:51:00.006-05:00

I recently read Sid Savara's article about how engineers can more effectively communicate with non-technical managers. I think Sid does a great job on the engineer's side with some really good insights as to how engineers think of communication. It's a great article that brings an honest (if somewhat naive) assumption that people simply want to communicate.

But it takes two to tango, and I'd like to challenge the assumption that a non-technical* manager can be an effective manager of a technology group with the right communication.

Let's use some simple logic first: Say it is possible to manage a technical group without being "technical". Then you are implying that management decisions essentially have nothing to do with technology; management is management - maybe it's about people, maybe it's about process, but it has nothing to do with technology in any fundamental way.

This leads to two supporting fallacious assumptions (which are very common in business):

ideally all management decisions in a technology group can be translated into non-technical terms. Any failures in this communication are ultimately the responsibility of your technical team to address (otherwise you'd have to learn something about technology, which you've already decided is essentially irrelevant to management concerns).
management issues are essentially simple and easy to act upon unless someone is making them needlessly complex and/or resisting doing any real work (i.e. your technical team).

Let's put it another way: Would you say that a non-sailor Captain could be an effective manager of a ship? Would you say that a non-diplomat could be an effective manager of an embassy? Would you trust a non-mechanic crew chief as an effective manager of a NASCAR race car team? Would you say that an untrained civilian could be an effective manager of a military unit?

Would you say that you can be an effective owner of a sports team without knowing anything about sports or the sports business?

No, because in each of these areas a knowledge of the specific problems and solutions is a requirement of the role of leader. Yet popular business philosophy has started with the assumption that all manager positions are essentially identical in their demands and execution.

It's time to start recognizing that technology isn't always simple and that effective technology managers have to be at least somewhat technical -- it's a requirement of the job.

* I'd like to point out that some managers consider themselves "non-technical" because they lack the domain-specific jargon or experience in a technology group. Maybe they were a biologist, instead of a computer scientist or at least they have some appreciation or experience with science issues. I consider these managers "technically capable" -- they are capable of understanding the arguments with some effort in learning or developing common terminology. This is a more amenable situation because these kind of managers don't make the fallacious assumptions noted above. They are eager to think through problems and find solutions even if they don't have the specific vocabulary. Given Sid's points of effective communication, they will rise to the very technical challenge of running a technology group.

iTunes is bad at eclectic collections

2009-01-22T09:26:00.004-05:00

My music collection is kind of eclectic (genre-wise). You never really know what you'll find in there... it might be Bach performed by Emile Autumn (Electronic?) or Angry Skies (Dance Pop?!)... ok so I noticed that the genres on a lot of the music are completely out of sync with reality. They seem to assign the genre label based on the artist rather than the individual song.

So I tried "TuneUp Companion," but this just made it worse. Now I have 57 very specific subgenres ("Electronica Mainstream", "Post-Modern Electronic Pop").. yikes! It made everything more specific, but now my music collection is fragmented along 57 branches in a completely alien non-intuitive way. The "Genius" feature isn't much of a help because it unfortunately ties into the same organization system to help construct it's playlists - "garbage in, garbage out" as they say.

So what's wrong with this? I know iTunes is just trying to leverage existing ways of cataloging music, but the traditional ways of cataloging were born of record companies and promoters, not listeners. Other contemporary software allows multiple tags so that you can capture all the different terms that apply to an item (such as a blog post).

iTunes needs to realize that there's more than one way to organize things in the world.

KillerNIC: trend or OS "epic fail"?

2008-12-15T12:48:00.002-05:00

I was following a friend's twitter the other day that turned me on to a product called KillerNIC.

This is a high-end gaming network card that advertises superior network performance in games. At first glance, I think there's nothing to this, after all, network interfaces have become commodity items for years now -- hardly anyone buys a dedicated NIC card that isn't part of a motherboard or laptop these days.

But then I come across the striking claim: "Completely bypasses the Windows Network Stack"

Wha-what?! Ok, part of me thinks this is really cool. Maybe its the beginning of hardware optimization of parts of the stack that were previously tightly integrated with a particular OS. DMA and asynchronous bus access is the future! Building-block integration stacks maybe?

But another part of me says "wow, windows has really epic failed to get to this point." Let's see, it was cheaper and more effective to create separate hardware and completely bypass Windows Network stack than it was to deal with the cruft of 10 years and bad plug and play interrupt models? Having been a Win32 developer... yeah, actually I can totally believe that.

Apparently this card also reigns in the "spoiled child" model of modern application behavior. Now when McAfee or Windows Update "calls home" for updates in the middle of your game, you won't be disconnected. The card can be configured to give maximum priority to your current focus application and idle-priority other background apps.

This points to another epic failure of application developers. It's gotten so bad that we simply can't trust you any more to do the right thing, so we're taking away your network privileges at the hardware level. Ouch! Once again, I can totally believe that.

Magpie: monetize your friends and family...

2008-11-02T11:07:00.004-05:00

Just ran across an article about Magpie on Mashable today. Magpie is basically a way to sell your friends down the river -- or in more marketing speak: "a way to monetize the popularity of your social network."

Maybe this sounds like a good idea to some bright people in marketing -- after all, it's viral, it has presence and it's easy. But why is a social network so popular to begin with?

What makes a social network work?

Let's start with the most basic social network of all: your family.

Now imagine that you show up to Thanksgiving dinner, but instead of hugging your mom, you tell her that she really should check out [brandname clothing], and that [brandname gravy] is much better than her homemade, and everyone should use [brandname toiletpaper] after the game, (brought to you by [brandname], of course).

By the way, after you tell your family all this, you mention sheepishly that you weren't saying these things because you really cared about them, but because you're getting $0.12 per impression.

Of course, your family (I hope) loves you, so maybe they'll put up with this, like so many families put up with Amway, candy-bar sales, etc. They'll think "oh boy, I hope this is just a fad"... if it continues for a while, you might find yourself getting fewer invitations to Turkey day each year.

What's the lesson?

If you take a shared resource like a popular social network and exploit it, you may end up weakening it or destroying it (i.e. Tragedy of the Commons). This kind of exploitation has already had a chilling effect on e-mail, instant-messaging, and blogging channels.

But there's a better way, if you really understand the value of social networks...

Corporations are already treated by the law as virtual people. Why don't you simply act like virtual people and start your own social networks instead of co-opting ours? If you are really cool/hip/interesting "people", then we will follow you! We'll tell our friends about you and they'll follow you too. You'll have your own community to say anything you want!

And you won't be pretending, faking, or buying your friends.

The dark side of subscriptions...

2008-11-01T13:24:00.005-04:00

I decided to do a little financial house-cleaning today and cancel any services that I don't really use anymore. It's not that they were bad services, or even malware scams, they are legitimate services from Fortune 500 providers... I just don't need them right now.

I signed up for them originally, because it was easy, I could do it online, it was fast, and I wanted that service. But as I found out today, sometimes they are virtually impossible to cancel once you've signed up for them.

Obstacle #1: How do I cancel my service?

Although the subscription buttons are easily accessible, it's hard to find a "Cancel Service" button anywhere on the account pages or in the company support pages. I found a cryptic blurb giving a phone number or an address to send a written request to.

I noticed that the cryptic phone number was different from the main service number. It had different restricted service hours (for example, only 10-4 M-TH, PST only) than the "purchase" side of the business which had 24 hour sales support.

Obstacle #2: Automated Call Centers

Many companies claim to have wonderful customer service records. However, "cancellation" was buried in the voice mail of this company, making it very difficult to navigate to the correct option. I was disconnected repeatedly while trying to cancel their service. Each time, I navigated painstakingly through a slightly different path in the system, was put on hold for 5-10 minutes, sometimes contacted by a service rep who told me I'd be forwarded to the "correct department" and then simply disconnected.

What kind of customer service is this?

Obstacle #3: Customer Service

Finally I got through to someone, who kept me in call no less than 30 minutes. 30 MINUTES! We went through all the options they offered, all my reasons for not continuing the service. I had to repeatedly defend my reasons for wanting to cancel the subscription. Why? Why should they even ask as a condition of "granting" my request?

In the 90's we saw the rise of the corporate-slave worker. Are we now seeing the rise of the corporate-slave customer? Do you have no rights to stop paying for a service you don't use!?

This is ridiculous and it has to stop. So here's my challenge to the industry:

If you make a subscription service easy to join, make it equally easy to cancel.

Otherwise you aren't being honest about your business model, you're not really keeping customers by offering a wonderful service that they use, you're merely keeping them by way of a corporate-protection racket.

Going forward, I refuse to sign up for any more subscription services unless they meet this simple criteria.

no-risk investments...

2008-10-13T19:24:00.003-04:00

I woke up this morning to an NPR segment featuring a UK professor calling for no less than a revocation of the 1997 Nobel prize in economics regarding option pricing formulae. At first I thought he was a bit extreme... now I'm not so sure.

The Black-Scholes method states a formula for certain strict assumptions that describes how to build a "risk-free" portfolio. Since this formula has been adopted by the Financial Accounting Services Board and widely used throughout the industry for predictions of options pricing it has garnered a fair amount of negative attention with the recent world economic crisis in the news. But is this unfair attention?

This news woke me up. I feel as though I must be missing something, like I've slept through the past decade of hedge funds, etc. never really understanding the mechanism... but I understand enough math now to at least comment on the apparently disasterous use of these ideas. Maybe I'm missing something, and a specialist can clarify why it's not really as bad as it sounds...

So, as I understand it, the basic mechanism is a clever way of observing the derivative of the valuation such that a hedge is perfectly placed so that either way the stock goes, the option will compensate and result in a magically "risk-free" portfolio. Obviously, this is huge news for anyone in Wall Street, since there is always substantial risk associated with every transaction.

Now, there are some known shortcomings with this model that economists were well aware of. For one, it makes certain strict assumptions about drift and volatility, namely that they are constant. Volatility is the more important concept here, because while it couldn't be known directly, there was a clever way of flipping the Black-Scholes around so that you could solve for the actual volatility given historical data. So quickly economists found that this value was anything but constant in the real world. In fact, they talk of volatility-spaces as a case of solution spaces that can have many shapes depending on the type of investment. But these are simple statistical generalizations -- there is no underlying theory of the shape of these spaces!! (at least not yet)

So, what we have is a system that allows us to concretely know the shape of the space AFTER it has happened, but has zero-predictive power to tell us what will happen. Now, economists apparently desparately wanted this space to have at least local-linearity, but keep in mind the essence of the space is unknown and is at least indirectly related to a geometric brownian factor (i.e. random!) -- so it's extremely unlikely that any linearity exists in the first place... but it's safer to say that we simply don't know what it is.

Now what mathematician in their right mind would suggest that this was anything more than simply shuffling one unknown into another variable and then claim that at least it is "somewhat useful as a first order approximation which can be adjusted"?

Are you kidding me? Based on what? Based on local linearity assumptions and cooperative historical data of the first few years?!? The space is unknown man! What made them think they could "adjust" it in any way predictively that made sense?

But it didn't even work that well when it was first introduced. Look at Merton's first board position at a hedge company that failed so spectacularly in 1998 that again the Fed was concerned about bailing out the industry. How did I miss that towards the end of the dot com bubble?! 4.6 billion dollars in four months!! (of course back then I'd never even heard of a hedge fund).

Yet the Financial Accounting Services Board adopted this method of calculating options value and now it's widespread use is implicated in the worldwide economic crisis you see before you.

Can someone please explain how this made any sense whatsoever?

[update 10/18]: Paul Wilmott's blog article in defense of Black-Scholes is an interesting read -- he's someone who does this for a living and so he has a much deeper view on it than I do certainly -- (it's a reference backing the wikipedia article and also provides additional references to check out)

He points out that Black-Scholes is very robust on the averages in spite of the obviously poor assumptions of the model, and therefore is a relatively simple model to use in practice (i.e. an industry workhorse even though it's not perfect). There are more complicated models but not a lot of evidence that they work any better. He asks why use more complicated models when the fat-tail isn't that important all the time?

Well, what about when it is?

Such as when the risk suddenly becomes very high market wide and the correction deltas are huge? Now you have a function that you assume is locally-linear (because of it's physics origins) -- but you don't. I think Black-Scholes might only be a proper "industry workhorse" when volatility is relatively constant.

It failed massively in 1998, and it failed massively now. Is it coincidence that the market was experiencing wide economic volatility at both those points in time?

I still have a big question about how worst-case scenarios were investigated on this model? If a handful of investments with high-volatility were considered in the context of stable market data, Black-Scholes might behave acceptably well on the averages. But was a "total market meltdown" considered closely?

a scary-deep idea...

2008-09-25T22:10:00.003-04:00

I stumbled across this very strange idea the other day while talking to a friend about science and religion... it's related to the post I made earlier about quantum observers and straightening out what it actually means to be an "observer" creating reality.

Assuming that "observers" can create reality (and this is what the SEED article suggests has been shown experimentally over and over again) -- I raised one of my biggest conceptual problems with the idea: given multiple observers, how do we all "agree" to create the same reality? If we don't create the same (or at least similar) reality, then how can we communicate about it at all? More to the point, how come the observers seem unable to shape reality by their will?

She took up this last point by referring to a modern theologist's view that we also can't walk on water because every cell in our body would have to believe it was true in order to succeed...

At first I thought how silly, but then maybe there's something to that... what if it wasn't just every cell in your body, but every cell, every rock, everything? I thought specifically of something I read of Feynman's talking about another aspect of quantum physics: that there is really no separation, no distinction at that level between particles, rocks, plants, people... stars... it's all part of the same oneness.

Then I thought of something really strange... wouldn't it be funny if all the experiments and the maths weren't wrong, just our narrow interpretations of the meaning? What if everything in quantum physics was pointing to the idea that there was and had only ever been One Observer infinitely observing Himself unfold?

...I paused and thought of the imagery of Buddhism, lotus petals and Mandlebrots infinitely shimmering in space and time...

Science can only take you so far, then you must dream...

history, math and misery...

2008-09-24T21:38:00.002-04:00

Oh, so let me tell you about my history class.. I had a weird revelation today. As we were going through all these philosophers and statisticians views on the industrial revolution, I began to think, "wow, all these horrible living conditions, etc. were the result of really poor models of business." The curious thing is that many of them were (and still are) mathematical models!

For example, with tons of surplus labor, the early industrial businessman seemed able to approximate labor as (people x hours) = value. As each person cost money in wages, they desired to keep that side low, while simply scaling the hours to 12 or 14 per day (children too). As time went on, they slowly realized that worker efficiency diminished past a point the more hours worked... so "counter-intuitive" to the earlier model, they had to factor in shorter work days with more breaks.

Another camp believed that you should only pay workers subsistence wages -- because if given more, they would spend it on "recreation" or "procreation" (ha!). Population was viewed as a fundamental problem because food was determined to rise arithmetically, while unconstrained population grew geometrically (the simple models of the time) -- hence the ideal was to keep the population low by keeping the poor working so hard they had no time for anything else!

Unbelievable!

But something about all this suddenly struck me as a set of problems that has been undergoing successive refinement over the generations. For example, Nash's equilibrium is a drastic improvement on Adam Smith that more closely approximates investor behavior... and in so doing, determined the behavior of a whole new generation of economists.

Business always looks so random at the low level, but high-up there is a definite pattern of improvement after horrible costs and destruction -- I hate to think it, but historically speaking, perhaps pain is the best teacher after all? It's startling to think of this like some big genetic algorithm that is slowly crawling through the optimal ways to run business and increase production. Still, it's also very exciting, because it means the better the mathematical models, perhaps the less suffering we need before we learn...

Through better math, perhaps we can save the world.

daydreams on a strange loop...

2008-09-18T20:46:00.013-04:00

I came up with these fanciful meanderings after reading Godel, Escher, Bach by Hofstadter a few years ago... I was thinking about it again today so I dug it out of some old linear algebra class notes. It's fairly crazy in the details, but the "dance" of it is fun... don't take it too seriously, unless you take it as a warning of the dangers of sleep deprevation mixed with studies in linear algebra...

-lk

The fundemental problem that Godel's Incompleteness Theorem describes is that any untrivial formal system is either incomplete or inconsistent. However, a super-system may be built around the first system that does completely and consistently describe it! The trick is, you have to go "outside" the system you want to describe in order to fully describe it (Hofstatder calls this a "strange loop").

The reason this isn't a terribly useful solution is that the super-system itself can be shown to be incomplete or inconsistent. In fact, no matter how many systems you layer on top of each other to fix things, there is always one outermost layer that suffers from this problem.

So, combining two other things I read about (Godel numbering and Hilbert spaces) :

What if it were possible to somehow compose an infinite number of formal systems such that they converge on a finite system? Could the outer "error" (required to describe inner systems completely) diminish wrt the total in a manner analogous to a limit? Could the limit then approach a system that is both complete and consistent in spite of Godel's incompleteness theorem?

(ED: this suddenly reminded me of a recent article I read about Negative Databases for some reason...)

Godel numbering allows any theory or formal system to be represented in mathematical form. So, it follows that there must exist a set of Godel numbers that correspond to math in Hilbert spaces.

Hilbert spaces are very interesting, because even though points in such spaces have an infinite number of coordinates, angles and distances measured between any such points are finite! Weird huh? (common applications are quantum physics, signal processing, fourier transforms, etc.)

(ED: heh, imagining extra dimensions reminds me of that great short story "Dreams in the Witch House" by H.P. Lovecraft... creepy!)

Since points in these spaces are vectors with an infinite number of coordinates, it seems reasonable that each point vector in this space could be interpreted as a Godel numbering representing a potential formal system.

Since there are an infinite number of points, there must be an infinite number of formal systems being represented... yet, it's possible to define a line of finite measure from the origin of this space to any coordinate point given an angle and a distance (using polar coordinates).

A line represents an infinite set of points... does this imply that a finite Godel numbering can somehow represent an infinite set of formal systems?

(ED: woah woah woah Tex, this is quite a leap...)

This raises even more questions:

What's the the representation of the differential term used in the limit?

Does the differential actually converge towards a finite value?

(ED: it just kind of ends here... I probably wised up by this point. Sorry, no breakthroughs!)

No REST for Web Apps...

2008-09-13T05:10:00.050-04:00

[update 9/20/08: actually, the O'Reily book "RESTful Web Services" does a great job of distinguishing 3 categories of services: REST-ful resource-oriented, RPC and REST-RPC hybrids. The last category is very carefully considered. The other thing that I really like about this book is the author's "whale-fish" simile when talking about superficial details like technologies used vs. architectural underpinnings.

My initial frame of mind when I started this post was based on several sites that I've seen promoting REST-ful apps that in my mind are clearly making "whale-fish" mistakes he talks about. They simply haven't grasped the underlying architecture of REST enough to distinguish the difference.

At the time I was worried that this was a general trend. However the O'Reily book shows that the leaders of this movement know the limitations and are applying them only to the places that make sense.

I highly recommend getting this book.]

Yesterday I stumbled into what became a vigorous debate through a simple assertion with my friend who is a Ruby programmer:

It is impossible to write a web application without using a stateful connection of some sort.

It's an interesting debate. I think there's probably a difference between what he is calling "applications" and what I'm calling applications, but let's tackle it by looking at a quick history of the web and why REST was so successful for web 1.0.

Pre-REST

Before REST, users had to connect to remote systems through a terminal session or a special client. This was ok, except remote systems could only support a (relatively) small number of connections before they reached capacity. Even worse, the connections were idle most of the time since users are generally slower than computers.

So an interesting idea was to make the systems "stateless" (i.e. sessionless), by allowing servers to handle only one request per connection instead of multiple requests. This solved the problem of idle connections and allowed a single server to handle thousands of requests from separate users. For static[1] web documents this was a brilliant optimization and thus the World Wide Web was born.

Cookies

Of course, almost as quickly as the Web became popular, people at Netscape wanted to build dynamic web documents -- documents that could change their content depending on who was viewing them. Their first web application was an online "shopping cart" that could support:

browsing a catalog
adding an item to a virtual shopping cart
checking out and paying for the item

But they had one small problem: REST.

With REST the user only gets to make one request and each request is completely independent of any other requests the user has made. It's like going to a market where they have magical shopping carts: every time you put something in the cart or try to pay for it, it instantly disappears! Clearly this was unacceptable, so RFC 2109 was born: a method of creating a stateful session using cookies.

However, with cookie-based sessions came performance and scalability issues. Web applications and infrastructure unavoidably became slower and more complicated than the static web had been.

Secure Sessions

Almost as soon as the first shopping cart web-applications were deployed people also started becoming concerned about the security of their transactions. The next logical step was proposed in RFC 2818. This time the security concerns required a new protocol (SSL/TLS) that was layered underneath HTTP (i.e. HTTP/SSL, or HTTPS for short).

However, this didn't save web developers from the complexity of managing session state, it only exacerbated it. Now we had a stateful application (using cookies) on top of a stateless protocol (HTTP) on top of a completely separate stateful session protocol (SSL). This added so much complexity to web applications that only the most demanding and wealthy clients (i.e. banking and commerce) can afford to develop and maintain such applications.

Most web architects avoid SSL intentionally, citing poor performance and scalability and complexity of management and deployment as key problems.

The Search for Simplicity

In the 1990's (the golden age of the early web) we didn't care about such problems, because the technology was new and exciting, the code was small, the problems were mere annoyances... and basically everything worked. However, by the end of the 90's, systems had become complex enough that developers started to rebel.

Web-fundementalism?

One of the directions we rebelled in is a kind of web-fundementalism (a return to basics): all this "state-management" was a bad idea in the first place, we should return to REST-ful principles that worked so well the first time. But what principles are we talking about? URL-spaces? Do people really understand which aspects of REST made the web 1.0 succesful?

As I asked these questions, I realized that most of the alternative[2] proposals were simply managing session state with different technologies-- instead of using cookies they'd use GET params, GET url-spaces or POST fields; instead of webserver memory, they'd use databases. They hadn't really changed anything.

Session management by any other name is still session management and is fundementally incompatible with the claims and assumptions of REST, chief among them the idea that such applications are still scalable and can support caching.

Let's explore this idea a little more since there is such resistance to it. Say I have a web application that allows me to rent videos... you might expect such an application to have a REST-ful design with the following types of urls:

http://videostore/signup
http://videostore/customers/larry/rented
http://videostore/customers/larry/cart/pay
http://videostore/customers/larry/cancel_membership
http://videostore/customers/larry/bill/09-17-2008
http://videostore/customers/larry/overdue/Big_Trouble_in_Little_China
http://videostore/videos/Harry_Potter/add_to_cart

On the surface, this looks beautiful... it looks REST-ful. But let's dig into some of those assumptions:

1) Below the pretty urls, is state-management. Suppose the urls are all being served by dynamically from a database. That means the web site has to read my information from the database and display it to me (but no one else) when I access a url below "/customers/larry". One article I read said the REST-ful answer to this is basic web authentication. But that isn't REST-ful since it's implemented with cookies! surprise! your session's back!

2) None of this is even remotely cacheable. When customers cancel or sign-up, the url space changes structure. But how can I cache a url-space that is constantly changing? The simple answer is I can't/shouldn't. The "horribly-complicated-billions-of-wasted-dollars" answer is: sure, you can invest in configurable (or worse yet, "adaptive") caching at multiple levels and spend the rest of your earthly days debugging it. I can hear the sound of a thousand web developers slitting their wrists even thinking about this... gee thanks.

3) One thing I read is that REST-ful web services are easier than XML services to deploy. It sure looks that way from the service author's point of view initially, but then some critical questions came to mind: how do you know the difference between the verbs and the nouns? I don't want the user to accidentally cancel their membership by just browsing the site.

Some people say that we should use the other two verbs in HTTP instead (PUT and DELETE). But how do you know that the client has PUT the correct format when you don't have any structural form that can be validated? Does the client just have to guess until they get it right?

I suppose the way out of that problem is to use some fancy AJAX and maybe some JSON object definitions, but now we're headed back into custom serialization territory. Haven't we been here before? Isn't this why people stopped using SOAP in favor of simple XML and why people stopped using XML in favor of simple JSON? Sooner or later features get added and simple isn't so simple.

Einstein had an opinion about simplicity, he said "make things as simple as they need to be, but no simpler". Do we really need to learn this lesson the hard way by repeatedly wasting billions of dollars in new technology cycles that have essentially been resolving similar problems?

Anyway, because #1 and #2 basically break all the claims that REST makes about scalability and performance and #3 points out that it's not easier, I think that the perceived gains from simply applying a REST-like architecture to web applications is mostly falacious. Certain kinds of web services, maybe, but never applications.

Fortunately, a handful of other people were already realizing this back in 2004.

State-management

The other direction we can go--we should go, is to finally accept "stateful" web applications as a given. Google Chrome, Adobe Flex and Microsoft Silverlight are all moving in this direction. REST has it's place, but so do stateful applications. It's time to recognize the architectural pros and cons of each and use the right tool for the right job.

In some sense, this is all obvious -- it's just hard to see it because the layers of technology have always clouded the argument considerably.

[1] It's interesting to note that Fielding, the author of REST originally thought in the context of information and media access-- he did not think of contrasting REST with RPC - a stateful technique. (from http://en.wikipedia.org/wiki/Representational_State_Transfer#Claimed_benefits)

[2] There is one notable approach that covers a certain subset of web services in a purely REST way. However, this is not an "application" in a traditional sense, it's limited to "lookup"-style services.

There is a very simple litmus test to see if your service is in this subset: Could you also implement the service less efficiently using ONLY static HTML pages? If the answer is yes, then it qualifes.

lrn2patch: an open letter to Apple and updaters everwhere......

2008-09-11T06:14:00.003-04:00

Look guys, I know you're probably busy creating the "next big thing" over there in Cupertino, but can you have one or two guys research the art of deploying software via patch files instead of downloading an entirely new 80MB image every time you deploy?

In fact, let's make this a global rant against the "Check Updates..." installers out there:

please realize that you are using OUR computers when you design and architect these systems. (i.e. the rude neighbor: "oh I'm sorry, were you using that?")
please realize that other pieces of software may be trying to update at or near the same time. (i.e. the flood: "Me me me..." "Me too.")
please realize that liberal use of "restart" semantics to complete installations may be magnified by the above. (i.e. the tantrum: "stop everything and do what I want... NOW!")
[added 9/25] Please. Please make your updaters run at IDLE priority... there's no reason that a system update should interrupt the user or lock up the system while the update is occuring.

I realize that Windows forces #3 more than most devs can avoid. (mac less so, and linux is almost impervious to restarts from installs). But aside from those exceptions, you should be treating our computers with respect. You don't own them or the networks that connect them. You barely own the licenses of the software on them.

Please take some time to respect your users. Thanks.

google's chrome is hot

2008-09-03T01:59:00.003-04:00

This is my first blog posting from google's chrome. There's been a lot of chatter about whether this is a good idea, so here is my two cents:

Q: Firefox is already a great browser AND it's open source. Why didn't Google just invest changes into this browser?

A: Chrome isn't simply another Mozilla-fork, it has new low-level system features and architecture. Firefox is a great browser, and it's pushed Web 2.0 far enough to expose some serious problems in the underlying design of Web 2.0 and the browsers that support it. Changing Firefox just isn't practical when there are this many changes... it's better to start over.

Q: "Separate Processes" per tab? Why is that necessary?

A: The browser and web based applications are hostile territory. You can't trust that applications developers are going to write friendly, clean, compatible code, so don't trust them. This sounds counter-intuitive until you realize that all the rock-solid architectures do this: IP works because it fundementally assumes that communication failure is NORMAL. Likewise, modern operating systems believe in process isolation. It's a "good thing"(tm) that Google is now doing the same for the browser.

Q: Do we really need another browser?

A: Yes. Application developers demand more of the web. We should have a fully integrated debugging stack. We should have protection against rogue or errant processes. We should give the user visibility into what happens on their machine. The current browser architectures address none of this. The closest extension out there is FireBug (and FireCookie)... but this is only a start. There are limits to what you can do within a poor javascript environment -- Google knows this because they are also web developers -- our pain has been their pain as well. So yes, it's absolutely necessary that someone with the resources step up and help solve this mess.

By the way, people have said the same thing about Opera for a long time, but the truth is that Opera pushed both IE and Firefox towards better CSS 2.0 compliance when it beat them badly on the ACID2 tests. Likewise, there are serious problems with current browser architectures that have not been addressed that Google is now addressing.

More competition is never a bad thing because it motivates exactly these kinds of changes.

Q: So great, now I have to design for another browser as well?

A: Unfortunately that particular problem will never go away anyway anytime soon-- we already have at least 3 major browsers (IE, Firefox and Safari). The truth is that a certain segment will always push the latest browsers and gradually the rest will either move towards compatibility or lose market share as those sites become popular.

Q: So why use it before it becomes popular?

A: Stability. With blogs, online banking, web mail, intranet portals -- it becomes very important that the browser not crash in the middle of edits. If browsers were stable this wouldn't be nearly as compelling. We rely on a nightmare of javascript to run our sites now -- shouldn't that be based on a stable VM and multiprocess model in-line with modern operating systems, not 1980's technology?

hostile metrics...

2008-08-20T12:56:00.005-04:00

While countering a blog post (#25) about men's involuntary glances at women, I raised an equally involuntary tendency of women to "test" men.

The primary goal of such tests seems to be to present a problem that appears to have a simple logical answer, but in fact has no correct logical answer (i.e. a trap). A simple example would be the notorious question "Does this make me look fat?"

I started thinking about evolutionary advantages and motivations for such behavior...

Assume that reproductive resources are scarce and they have high cost and intrinsic value for both sexes. Anytime there are scare resources worth having, there are at least a few general success-strategies: fight, lie or wait.

Let's focus on "lying" as a strategy. Imagine a selection environment rich with liars who are trying to misrepresent their true value. Women who can't detect the liars from the nonliars are selected against evolutionarily speaking. So the goal is to find a metric that can distinguish liars from non-liars. But how could we do this?

We can't test with simple logic, because they'll lie ("Are you a liar?" "No. Of course not."). But what about attacking the premise of a lie instead? The general premise of a lie in this context would be to please the prospective mate.

So here's an interesting idea... frame the concept of a "test" so that any logical answer or pleasing answer by definition fails. Now, apply the "test" to prospective mates and watch the reaction according to the following:

1) If the male responds with logic, there is always a negative answer that will assert he is wrong, i.e. this allows the test to be chained -- continue testing until he doesn't.

2) If the male ignores the test, he's not interested or not attentive enough. This lowers confidence -- test again.

3) However, if the male doesn't ignore the test, but responds without logic (non-sequitor, humorous response) in an unexpected but interesting way, this is interesting. He's potentially demonstrated that he is attentive, but is not saying things simply to please his mate. This raises confidence -- test again.

What's interesting about this idea is that it doesn't define an end. This is a continual testing process that merely raises or lowers confidence. It can adapt over time. Of course, I don't know that women really use this process, but it is an interesting hypothesis.

entropy...

2008-08-12T08:05:00.000-04:00

The Second Law of Thermodynamics says that entropy always increases. Ordered systems are typically lower entropy (like a clean desk) and classically, they are said to have more available energy... so the 2nd law has always been a little sad.

However, what if there is a loophole? What if we:
1) look at increasingly complex systems of order. initially, these appear random to us, but understanding order in chaotic systems gives us more leverage to
2) do more with less. so we need less energy, thus creating more "headroom" -- does this (relatively speaking) reduce our apparent entropy?

Of course, it still seems like there is a trend of diminishing returns, (this just lengthens the tail in some sense) but now instead of implying a simple "heat death" of the universe, the 2nd law implies the rise of greater and more subtle organization as well.

Interesting.

the return of intrinsic computer languages!

2008-08-04T08:04:00.000-04:00

I recently found out about Processing. It's amazing. It's simple. It's portable.

It is the first intrinsic computer language in almost two decades. Languages for some reason have become cumbersome extrinsic behemoths... what do I mean by intrinsic and extrinsic?

Let's say you want to print to the console:

print "hello world";

This is a function that is intrinsic to almost every language in history. It's intrinsic because you didn't have to specify any concept of bindings to an IO channel, unicode character string loading into memory, etc. etc... you just do it.

In contrast, here's an extrinsic example of how to print text to an OpenGL surface. Notice how much boilerplate you have to weed through to finally implement basically two steps.

Now let's look at a similar example done in Processing. Wow. It's so much easier. The code focuses exactly on what you wanted to do... all the other setup and infrastructure has become intrinsic.

Back in the old days, most computer languages were intrinsic. You simply couldn't afford to write millions of lines of code to get something done. Fortran, BASIC, Pascal, Smalltalk... somewhere along the way this viewpoint shifted.

C, C++, and now Java have led an ever increasing charge towards extrinsic declarations. It happened for the best of intentions: we wanted flexibility. C wanted to keep the kernel of intrinsic functions small so that it was portable and powerful. For a time this approach was also simple.

There is nothing simple today.

In the current age we are deluged with extrinsic code. We are drowning in the most convoluted syntax man has ever created. Hibernate, Flex... the list of modern application environments and their dozens of sub-languages requiring explicit coordination has grown out of control.

And none of it is what we really wanted to do in the first place.

It's time to return to intrinsic languages. I'm not advocating a return to simpler non-portable environments that C and it's ilk freed us from. I'm saying the infrastructure has to get smarter -- it has to let us be portable, flexible, free from deployment tyranny, but it also has to be more intrinsic, transparent when we need it to be, opaque when we don't, subtle -- NOT obfuscated!

Processing is the most refreshing step in that direction since LOGO.

web design still sucks: an open letter to the W3C

2008-05-16T18:57:00.000-04:00

For many years I've followed the public line of the W3C: you should separate concerns, keep the html close to the data and use CSS for the presentation. Like other web developers, I've struggled between the right way to do things and the way that works (even when the experts don't agree what the right way is-- google CSS 3-column layouts for example). I've cursed browser makers for the seemingly arbitrary rules they use to render margins, box-models and standard controls. And I've struggled to understand the vaguest definitions of resolution and weighting models ever written.

But then I realized the other day, that the W3C simply lacks expertise in presentation-side implementation. At first this might seem like an outrageous claim but consider that there are two other standards that are at least as old as web standards that get this right: OpenGL and Postscript.

Both of these standards have a predictive presentation model -- if you say draw something at a given coordinate, you get something at a given coordinate, on any opengl hardware in existence, on any postscript printer in existence. There's no crying about "relative" units either -- OpenGL units are completely relative, but they are superior to W3C notions of length because they are consistent! It's a far cry from an industry that can't even render their own standards correctly or consistently.

So why can't web standards give us the same predictability? I think it's because the W3C is fundamentally confused about it's purpose and goals. On the one hand they wish to promote the idea of the semantic web (which is primarily a data-centric view), and on the other they want to begrudgingly hand some presentation layer scraps to visual designers who do the most heinous things imaginable to get their work done. I say "begrudgingly" because CSS was only proposed when table-layout threatened to blot out the semantic web forever. I still get the impression that the W3C views presentation concerns as an annoyance that interferes with their data-centric preferences.

I know there is some good work being done in CSS3 and HTML5 that we have yet to feel in the mainstream site design (and even when we do, we'll have to consider the mishmash of currently deployed browsers to make it work correctly), but it's too little, too late.

The W3C's biggest nightmare is already happening, people who care about consistent, quality presentation are turning to rich internet application frameworks such as Flex and Silverlight because it's just so much easier to use. These platforms solve the browser problem by completely bypassing it -- the plug-in suddenly becomes the new browser! They can even back their presentation with xml datasources as well, so other clients can choose to view the pure xml (which is even better than XHTML for semantic purposes since it doesn't need to be cluttered with presentation cruft).

a philosophical take on quantum realism

2008-05-08T11:28:00.000-04:00

I got the latest issue of SEED (#16) and was instantly captivated by the article on p.50 "The Reality Tests". The article goes into one of the more controversial questions of quantum mechanics, the idea that particles might not really exist until they are observed. Many physicists (including Einstein) have disagreed with this idea because it is so alien when compared with our everyday experience (i.e. the moon doesn't cease to exist because we stop looking at it, right?).

But the article explains that new experiments disagree with the idea of "realism", (i.e. that particles have an independent existence before measurement) by more than 80 orders of magnitude. So are we left with the unavoidable conclusion that we can create reality simply by observing it?

I'm not a physicist, but even without physics there are some philosophical problems with such a question.

First, in its most basic form, the explanation that reality is created by observation isn't an explanation. It simply pushes the problem into a different space: "observation" and what it means to be "observed".

I'm not aware of any accepted rigorous definitions of those terms, however the ideas of an observer "creating" reality seem to unavoidably involve free will. After all, it's commonly implied that the observer gets to choose what to observe, right? Except we don't really seem to have conscious control over what gets created "in the large". (For example, I would like to observe a million dollars in my living room right now... nope. didn't work.)

Maybe we don't have conscious control over our observations... but then we have to explain why observers create the same (or even a similar) classical reality that we perceive as coherent when theory says there could be an infinite number of alternate classical realities that are equally possible. Why this particular one? Collective unconscious?

If we disregard a "collective unconscious" of some sort, we are left with something even more dramatic: for some reason, the science of observations collapse on the same reality regardless of all the possibilities -- i.e. you are drawn back to a system of fate without any free will.

But a deterministic system by any other name is still a deterministic system. We started out wanting to say that the particle had no particular existence before observation and instead ended up with a system where observations are as deterministic as the previous definitions of the particles they tried to explain.

So, really, when we're talking about "observer" and "create", we are talking about something very special that doesn't match our everyday meanings. We should figure out what those terms mean before using them in quantum theory.

p.s. there is another wrinkle with "observer" -- quantum physics says you can't observe things without affecting them, so at the simplest level it sounds like observers are just participants in interactions. But interactions affect all matter and energy, so does this mean that everything is an "observer"? If not, what is special about "observers" then?