My (very shallow) thoughts on Dart

Being the language nerd that I am, I actually find it fun to learn new programming languages. Now typically this is nothing more than me reading all of the official documentation and writing some toy examples that give me a very shallow, quick-and-dirty feel for a language. Since I have been involved in language design for nearly a decade (started participating on python-dev in June 2002) and have done toy examples now in 18 languages (17 actually still run; I have never bothered to get Forth to work again after a gforth change broke my code), this is actually usually enough for me to grasp the inspirations for a language and thus understand its essence.

At work I have been doing some JavaScript work for an internal Chrome extension and dashboard and so that led me to want to look into what Dart had to offer over JavaScript. I know the language is only at version 0.09 (and still changing weekly), but the fundamentals are there so I wanted to see what the general feel of the language is (and will continue to be).

I also know Dart is somewhat controversial for some people. Personally, I fall on the "competition is good" side of the argument, not the "OMG fragmentation" side. I want ECMAScript Harmony to still happen and give me a cleaner, tighter, more functional JavaScript, but that doesn't mean Dart doesn't have a place in the world as a cleaner OO language for the web. Besides, me thinking otherwise would make me a massive hypocrite as I began working on Python before it was cool (I feel like I need a hipster meme for that statement, but I digress) and I have worked hard to convert people to Python from other languages. Hell, I have tried to foster competition between the Python VMs to get them to push each other to perform better and be ever more interoperable. IOW I don't totally buy this fragmentation argument.

Going into learning Dart I knew who was involved with the language which is what will inherently define how a language feels. I knew Lars Bak of V8 helped design the language, which meant it would have some design restrictions put on it to make it have a damn fast VM. Josh Bloch has been helping to design Dart's library which meant some JDK feel to it. I also know Jim Hugunin is involved which should also help with the VM speed. So fast with an API designed like the JDK.

What did I find? A language with a damn fast VM and a standard library that felt like the JDK. =) Take OO as a Python programmer would expect (e.g. pure OO where everything is an object, not dogmatic OO like Java where everything has to be in an class definition), make types entirely optional for testing and tooling purposes but enough support to use interfaces and generics, and then toss in abilities based on what JavaScript allows and then you have a good idea of what Dart offers.

So, Dart has optional typing. In case you have not heard, Dart does not use type information at runtime for performance and only throws any form of fit if a type doesn't match what is specified unless you run in checked mode. If you do that then you get warnings about possible type issues. But Dart's type system is unsound so don't expect typing to catch every error that a more strict type system might even when you run in checked mode. Dart views types as helpful documentation and a way to help tools assist with things, period. I actually find it rather refreshing to have a language that treats types as just documentation since that is really what they are for the programmer (VMs can use it for performance, but it isn't required for good performance and type safety only saves you from a minor set of bugs which every Python programmer probably realizes eventually =).

But that's even if you bother with types! You can write all of your code without types and everything will run without issue. Even generics are optional, so you can declare a function accepts a List or List; Dart doesn't care either way and it alleviates covariance/contravariance headaches by not caring if you don't care either. It's actually rather nice to have non-library code be written quickly using dynamic typing and only add in the type information for library code where you care about what interface is expected. IOW I think Dart strike a nice balance with how it does typing and I actually feel fine using types when I know what I expect to accept in my own code that I don't expect anyone else to rely upon.

Dart is OO, not prototypical like JavaScript. It's single-inheritance, which I'm fine with. It does have interfaces as one would expect in a statically typed language, but it softens their expense by allowing one to define a default implementation of an interface. What this means is that the Map interface will also give you a HashMap instance if you call new Map(). I suspect they snagged the idea from Scala  where you have the Map class which hides HashMap from the user if you simply don't care about what Map implementation you use.


It does have a modicum of privacy by using a leading underscore for signaling something is private, much like Python. But the privacy is enforced at the library-level or is public, period. Every field automatically has a getter and setter defined for them, so there is no way to force a private field (which I think is a good thing since I find private privacy bloody annoying). I also like that getters and setters are directly supported by the language with automatic generation show you don't ever have to see a setSomething()/getSomething() function call just to read/write a field, but you can do something like Python's properties very easily.


The standard libraries are fine and just feel like the JDK. Things are very much LBYL rather than EAFP. I am willing to bet (although I have not tested this) that exceptions are a little expensive in Dart (since exceptions are hard to optimize) and so they would rather go the LBYL way. But they still went a little overboard in my opinion on some things (e.g. the list interface has a last() method instead of supporting negative indexes). But there is nothing there that is making me run away screaming.


One place I do think Dart could use some improvement is simplifying their constructor rules. Upfront Dart has some nice syntactic sugar for a construction where you directly specify how a constructor's arguments map to instance fields, avoiding having to declare the constructor parameters and then also write an assignment. OK, I like that.


Dart also has initializer lists which let you initialize final fields. OK, that's cool and a nice idea taken from C++.


Constructors are not inherited. OK, that's fine since you probably want to be explicit about how you tweak stuff. But there is an exception about the default, no-argument constructor calling the superclass' no-argument constructor. So while not technically inherited, it might as well be in that single instance. And all defined constructors will automatically call the default constructor, which if it isn't defined you must explicitly call a constructor somehow (probably in the initializer list of your constructor). Um, OK...

And you have named constructors. This gets you around from the lack of type-based method overloading for constructors. OK, I can go with that.

You also have constant constructors since fields can only be initialized to compile-constant values. Fine, that's for performance and determinism in instance creation, so I can grasp the desire for that.

And then you have factory constructors. OK, this is where I go "WTF people". This is so that you can have a constructor that actually doesn't create a new instance but instead can return something else other than a new instance (think of Python's __new__() or any of Java's static factory methods). But this lets you use the new keyword on a factory constructor instead of using a static method. And that to me seems unneeded.

So lets recap what constructor options we have. We have regular constructors, default and defined, which supports initialize lists. You have named constructors. There are constant constructors. And you also have factory constructors. If you don't count the default constructor as special that means Dart has four types of constructors. WTF!?! I realize that Java's FactoryFactoryOfFactories crap has probably spooked the crap out of the Dart designers, all the while having Java influences making them think they need the new keyword for anything that would return an instance of a class, but this seems a bit much. Dart's function definitions are rich enough to allow for optional arguments, etc. which would suggest that the typical constructor can do the job of named constructors with static methods picking up the slack where absolutely necessary where factory constructors are used. Maybe I'm missing something here, but I think they tried to design for everything that is bad about Java's constructor mess without stopping to think what their function definitions already buy them, all while making sure the new keyword was used.

Luckily that is the only bit of Dart that I found poorly designed. Everything else is reasonable and something any JavaScript programmer will be somewhat familiar with or quickly grasp.

Now as I said, I only did toy examples in Dart beyond reading the docs from beginning to end. If I had more time this weekend I may have done one more coding example that was more involved, but I ran out of time. But based on what I have read and what I learned, I am happy with Dart and would be content in using it for programming for the Internet. I would also be totally happy being asked to use it in a situation where others wanted to use types (e.g. I would be fine ditching Java for Dart if people really felt the need to hold on to their types).

Thoughts on using function signatures as a DSL for CLI parsers

I have no idea why, but this morning I thought about a decorator for delineating what function should be treated as the main function (e.g. using a decorator instead of the traditional if __name__ == '__main__' idiom). Now I solved it in my head on the spot, and then immediately realized someone had to have solved this already. Turns out various people have done things as nuts as examine stack levels to detect the __main__ name, but the most straight-forward solution I found doesn't do anything nearly as nuts or CPython-specific and is basically what I came up with. There was a red herring, though, in everyone's solution where they claim the decorator has to be on the last function in your module. While technically true when using the decorator as a decorator only, you can also just as easily not decorate the function and instead, at the end of your module, do something like main(func) since that is the same as decorating func with main.

A really simple expansion of this idea of helping out with defining what function is the main function, is to pass in sys.argv and to return a value to signify exit status: sys.exit(func(sys.argv[1:])). So now you have made the decorator more useful than replacing the old __name__ idiom.

But while that is nice and helps deal with the very common case, I wanted more. Why can't you introspect on the arguments the function takes and use that to automatically generate a command-line parser? I did a search and the best I could find is entrypoint, but it doesn't go far enough for me. What I want is to use the full expressiveness of function parameters in Python to express as much about what should/could be given on the command-line along with passing in as little as possible to the decorator in order to replicate the common case of command-line parsing; think just as easy as getopt but more powerful by using as much of argparse as you can without coming up with complicated rules about how things should work (since once you pass a certain complexity threshold you should just build the argument parser using argparse's API directly and stop trying to optimize for it like I'm suggesting).

So what do we have at our disposal to build such a decorator? We have positional arguments so we know how many arguments are required without some specific qualifier. We have variable positional arguments (e.g. *args) to take an optional number of extra arguments at the end of the command-line. We have keyword arguments which are optional flags that one can specify. You could even have variable keyword arguments for major flexibility, but that just seems like a total lack of structure the CLIs just don't typically provide. With all of that you can reproduce getopt without any issue for long-form names. For short names, I would say you need to pass in a mapping of short names to long names into the decorator. Same goes for long names to help string (you can use the function's docstring for the main help for the app itself).

But where things get really interesting is when you take into consideration function annotations. That opens up the possibility of going beyond getopt and potentially supporting argparse's action, nargs, and type options. Take the type option as an example. You could say limit:int=10 to have a command-line option called --limit which only accepted an integer and defaulted to 10.  This obviously could also work with float or any other type where you can just pass in a string to the constructor to get back an instance of the type. So you have a general case which can be useful, but you can you potentially special-case some things to get enhanced functionality where it doesn't make sense to simply take in a string?

Lists pose an interesting option as argparse provides both nargs for specifying the number of arguments to a single option, or the append action for accepting multiple instances of the same option and accumulating them. In my mind both can be expressed in a way that I think makes sense but some might view as too magical. If you specify names:list=[], then that supports the append action, e.g. --names Brett --names Andrea leads to names being set to ['Brett', 'Andrea']. But if you were to do names:['+']=[], then that would get the same result from --names Brett Andrea. In other words, the list type specifies the append action while a list instance specifies using the nargs option with the single item in the list acting as the value to set to nargs.

For booleans, I would want the use of the bool type to mean use either the store_true or store_false action based on what the default argument was. So turn_on:bool=True would use the store_false action since the argument is meant to be a boolean and it's default value is True, meaning that if the option was specified it represents the reverse.

Finally, the tricky bit is for files since that is a common command-line argument and you might as well open the file and close it for the function. The solution argparse uses is a specific FileType class where you can pass specific arguments to use when opening the file. The problem is that it doesn't support everything open() does, e.g. encoding. So what I would want to do instead is provide a partial function that took everything but the file path and then when it came time to call the main function, passed in the file path to the partial function, passed the returned file to contextlib.closing(), and then passed it on to the main function. You could even generalize a lot of this and simply say that whatever is specified as the function annotation, if it isn't a special-case like lists, then you call the annotation with what came from the command-line and if it provides a context manager it is used before calling the main function.

So those are my thoughts on using function parameters as a DSL for getopt++/argparse-- functionality on a Saturday morning. Honestly the most complicated bit would be constructing the arguments to pass to the main function in the right order, otherwise it's just introspecting on a function's parameters and making the proper call to argparse. But then again the real question is whether anyone thinks this at all sounds reasonable enough to code it up.

Playing with the Ninja build system

Whenever I learn a new programming language I end up writing some toy examples to try to get a feel for what the language is about. This leads to the need to build code using many different compilers with their own flags, quirks, etc. Up until today I had used SCons for my build setup. But honestly, it always seemed like overkill to me. Because I only had about 5 programs to build per language with at most two files used to produce the program, a full-blown build system was never really needed. Add to the fact that I am building for languages that no build system would have built-in support for, it led me to always have a wandering eye for another build system I could use.

This past week someone on Google+  shared a post comparing configure+make, cmake+make, and cmake+ninja. I had never heard of Ninja, so I decided to have a look. It turns out someone had written a build tool whose only explicit job was to take a DAG, figure out what needed to be built, and then execute the commands for the build. No crazy metadata checks like Make, or fanciful features, just bare-bones building. Ninja was actually designed to be a target for other higher-level build systems like cmake which can do the pre-computation of what the DAG should be, leaving it to Ninja to drive the needed compilation.

What attracted me to it was that it was fast and the syntax was simple. I have code examples for 16 languages, of which 10 have build rules (one happens to be Python 2.7 as I pre-compile the .pyo files). Turned out to be a pretty straight-forward process to take my custom SCons commands and just translate them to the subsequent shell commands that Ninja would execute for me. They are a tad verbose in order to make sure that the ninja -t clean command would clean up all intermediary files (I'm looking at you OCaml, Haskell, Java, and Scala). But as I said, I typically never have more than 5 programs to build per language, so it wasn't that much of a burden. And if I really cared I could have written a Python script to auto-generate the Ninja files for me, but I decided the effort of writing the code would be just as much as writing the build files by hand.


I realize I could have used Make, but I honestly am not enamoured with that tool; requiring tabs just rubs me the wrong way. Plus it's rather slow in the common case of only changing a file or two compared to a complete build from scratch.


Overall, for my weird case Ninja worked out. For something more complex, though, I will consider looking at cmake+ninja as a build solution.

The re-launch of py3ksupport!

The reason the past few blog posts I have written have been App Engine-themed is because I have re-launched py3ksupport! I did a complete rewrite of the code to make it more efficient (since I'm paying $9/month for the site) and at the same time moved over to HRD so as to guarantee the site is always up.

Before I discuss some unique features of py3ksupport, I want to point out that right now that 56 - 60% of the top 50 projects based on downloads of their latest PyPI release support Python 3. The reason for the range is that some projects had in-development support last time I looked and since that can change underneath me I wanted to cover the possibility the data was stale. But the key point is that 8 of the top 10 projects support Python 3 and one of them has support under development along with over half of the top 50 projects.

So I'm sure the site explains itself (and the FAQ fills in gaps), but I figured I should explain some of the more unique features of the site. One is the metadata rating given to each project. Basically I wanted to shame project owners into updating their project metadata. LOTS of projects don't bother to specify the Python support metadata for their projects which makes my life difficult and is unfortunate for users. For instance, of projects bothered to specify the exact versions of Python they support then users could easily tell from the Cheeseshop (nee PyPI) whether they could use the project based on what version of Python they are tied to. I might have to do some public shaming at PyCon if the situation doesn't improve. =)

The other key point is that I personally keep the front page up to date. If a new project shows up on the front page that does not have the proper metadata specified then I personally search online to find out the status of the Python 3 support. Since I get emailed each day when this happens the situation tends to get fixed that day and will be noticed within an hour of me fixing it (the length of time I have things cached).

I'm hoping to eventually move to measuring an project's popularity based on the download rate for the lifetime of the application (instead of just the latest release) to give a more accurate reflection of how popular a project is.

How I bootstrapped importlib

If you have been reading this blog over the past five years I am sure you have read a post or five about my desire to bootstrap importlib into Python as the implementation of __import__. Well, as of today I'm willing to say that the difficult technological hurdles have been scaled! At this point the only thing holding me back from taking my code from https://hg.python.org/sandbox/bcannon#bootstrap_importlib and making importlib drive import statements are some small compatibility issues, integrating into the build process better, a code review, and python-dev sign-off. In other words all of the interesting problems have been solved, so I'm finally ready to write a blog post discussing how I pulled off what I have.

So how exactly do you import __import__? To begin, as with any bootstrap challenge, you need to figure out what is available to you so you know what your design parameters are. In my case I knew I couldn't import anything that required filesystem access since half of import is handling the search for a module (the other half is the actual importing); if I wanted to import a file I would need to essentially write half of import in C to work properly. This restriction also has unexpected side-effects, e.g. you can't rely on open() because that is part of the io module which is a Python module.

That meant I could only rely on built-in modules. If you run sys.builtin_module_names you will discover what is available directly within the CPython binary. The question then becomes if that is enough? It turns out that yes, those built-in modules are enough to perform an import. OK, so you know you have the bare minimum modules required to do an import, but how the heck do you get the built-in modules into the global scope of the module that imports module since you can't use an import statements?

This is when Python's dynamism comes in handy. Since the import statement doesn't do much more than pull in the module object and assign it to a variable at the global scope of the module, I just needed to get the module object for importlib and assign to its __dict__ the built-in modules I needed. Turns out that sys and imp are enough to allow importlib to handle the import of the rest of the built-in modules needed for import to work, so that kept this bit of code short.

But this brings up the next quandry: how do I create a module object of importlib? If I end up searching for importlib on sys.modules then I would have ended up implementing a decent chunk of import itself. So how could I get the module object? This is when frozen modules comes into play.

A frozen module is just a C array containing the marshaled code for a module (which is what a .pyc file is sans magic number, timestamp, and now file size of the source). Since marshal is a built-in module then frozen modules can be loaded without issue. That means you can load a frozen module without using import (much like importing built-in modules).

And that is all of the parts needed to import importlib w/o import. =) To summarize, you get importlib set as __import__ by doing the following:

1. Import the frozen module (i.e. read in a C array of a marshaled module object and unmarshal it)
2. Import sys and imp (built-in modules, so done in C code by calling key C functions which return module objects) and set it on the module object
3. Call Python code to import the rest of the built-in modules using sys and imp
4. Set Python-based __import__ on the builtins module

And voila! __import__ ends up implemented in pure Python code. Now I just need to clean up the code, fix the compatibility issues, rip out the old C code, and get python-dev to sign off. =) Hopefully I will get far enough I will have a lightning talk at PyCon with benchmark numbers to show this is actually all a good thing (including ripping out a ton of C code, especially if I can re-implement chunks of imp in pure Python =).

Asynchronous XML-RPC in Python

Do you use XML-RPC (and specifically the xmlrpclib/xmlrpc.client from Python's stdlib)? Do you like multi-calls? Wish you could construct your XML-RPC multi-calls in a way so that you could make them asynchronous by constructing the call from scratch? Then you're in luck because I already did the hard work of figuring out the details for you! =)

Grab bag of tips when working with App Engine unit testing

[update: App Engine doc bug is being fixed and should be publicly visible in 1.6.2 or 1.6.3]

First, an announcement: register for PyCon! It's a great conference and tons of fun. Early bird ends on the 25th.

With that out of the way, this blog post is going to be testing an App Engine app. This can be, liking testing any complex system, is painful. And unfortunately the documentation on this subject is somewhat lacking for App Engine. But hopefully this post about random things to be aware of when writing unit tests for App Engine can help prevent someone else from having to learn the hard way like I did.

Working with App Engine backends

This post is all about backends in App Engine, but first I want to make two personal points.

One is that I'm sorry I have not been updating this blog (or really contributing to Python and its great community) more this past year. It was a rather insane 2011 for me personally: I finished my Ph.D., got married, moved to SF  (w/o my wife thanks to immigration), started working at Google full-time, moved to Toronto (yes, I moved internationally within the span of 6 months), transferred to the Google Waterloo office and started on a new team. In other words I have been stressed out and busy and busy continuously for what feels like ages. But now that I am back with my wife and I am done with the crazy international moving I don't expect 2012 to be anywhere near as nuts (nor any other year in the near future for that matter if I can help it).

Two, I am no longer on the App Engine team, so all that I am about to say (and will also say in the future as I have a couple of posts to write on App Engine) is from me as just another person using the service. I have been working on a redesign of a website of mine in my spare time and these next couple of posts are based on that work (if you following me on Google+ you know what it is, but I'm not ready for a full-blown Internet debut so I'm not going to link here yet). I don't have any insider knowledge here nor speak for the team.

PyCon 2012 CFP ends in a month!

The title says it all: the CFP for PyCon 2012 ends in a month, so you should seriously start thinking about getting those proposals in! As I say every year, you do not need to be a Python celebrity or  have presented previously in order to get a talk accepted. Having been on the PC from the beginning of PyCon I can tell you that we very happily give slots to people we have never heard of. As long as your proposal shows you are organized and your topic is interesting your talk will be  given a serious chance for acceptance.

And sorry about the lack of posts. My Python free time is limited at the moment as most free time is spent talking with my wife over a G+ Hangout thanks to US immigration perpetually dragging their feet, keeping her in Toronto and me in SF and causing general misery spread across the continent. That means what little time I do have I spend on trying to get importlib bootstrapped (which you can follow in my sandbox/bcannon repo in the bootstrap_importlib branch) instead of blogging. But once the bootstrapping is done (or I come up with some rather clever thing) I will do a technical post.

How to import a module from just a file path

Dr. Brown asked on Twitter whether there was a single expression (e.g., no semi-colons or abuses of and or or) that could import a module from a file path, either through a stdlib function call or just constructed from scratch; mod = import_from_path('some_file.txt'). Because it involved import, I got cc'ed on the tweet while various people tried to come up with a solution. In the end people realized that it's not possible in Python 2 (but it is in Python 3). But how hard could it be, right? Right?!?