DBIC::API

With work deadlines, holiday baking, and playing in the snow (in Texas no less), I got sidetracked and never wrote the advent article I promised for the Catalyst folks. I feel horrible.

As if that wasn't bad enough, apparently there are more complex configurations out there that I didn't account for in my DBIC::API branch (that is now merged in and is trunk), so it breaks people's apps.

So I know what I will be doing next week: fixing the brokenness.

Short post. See you next year.

Of more nerdhackery

I admit I am a dork. So I recently started playing Magic: The Gathering again with real-live cards. And it is rather a pain in the ass to figure what I need to build a bad ass deck with out shelling out for some stupid guide that lists all of the cards in a set.

Luckily, someone was awesome enough to type up all of the cards in various sets (including the activation text, if applicable). But you can tell the goofballs manage this data as a flatfile of text. It is very regular though. If only that were a database...

So step one is to build a schema using DBIx::Class that represents the various fields present in the file (with some of them broken down further so you could separate things like power and toughness instead of being a single field). Next, write the line-by-line parser using simple regex. Convert parsed data into created rows. And like magic, we have a database.

So what can we now do with this database beyond the boring crap like searching for cards by casting cost? Write a deck analyzer of course!

I haven't executed this part yet. But I will. And it will be most excellent. See, I figure I can take this data and throw rules engine at it. Provide a Cat app for building decks and running it through the rules engine to determine things like "Not enough land to support this deck."

Doing a quick search on CPAN returns a number of rules engines to use, but I think FSA::Rules shows the most promise. It is simple enough to not get in the way, and lets you define your own internal data.

The Cat app could obviously do more like build out dumb starter decks based some rules too. So far I haven't found any thing on the internets that does what I am wanting to build. This is good.

Hooray nerdhackery!

DBIC::API Hackin'

I've just finished up my branch of Catalyst::Controller::DBIC::API. What a beast. It is functionally complete and backward compatible, but I will likely continue to tune it, write more tests, etc.

For my work project, I was bumping up against some pain in the ass things when trying to make DBIC::API play nice with Extjs's grid stuff. I needed some more configurability in what the data looked like going to and from the server. This included properly handling JSON booleans as well.

So I embarked on an ambitious task to add those features in, plus Moosify it. I also wanted to do more validation up front before ->search() was ever called. You wouldn't even know if your ->config() parameters actually fit your ResultSet until the first request came in and you watched it bomb.

The validation part took the longest as it involved the most yak shaving. The outcome from that though, is Data::DPath::Validator. Making use of Yuval's wonderful Data::Visitor, I wrote a parser that generates Data::DPaths, which are then used (ala Data::DPath->dpath) to validate data. So you feed it a template (replacing the allowed, variable pieces of the data structure with asterisks) and it will create all of the valid paths for the template, then feed it a data structure to see if it is valid. By default it operates in loose mode so only one path needs to succeed for the data structure to be valid, but you can easily flip it into strict mode where /all/ paths must pass for the data structure to be valid.

What this tool gained me in DBIC::API was a way to verify everything from search parameters, to prefetch parameters based on the provided config options (which were templates). I admit that my tool was more general purpose than the limited use in DBIC::API, so I had to subclass and flatten some aspects of the generate DPaths.

Beyond that kind of validation, I shifted goodly chunks of the request munging stuff into its own class with triggers for validating against the schema of the model. So most of the aspects of validation happen in one place, at one time, and very early in the request.

Anyhow, you can check out branch of DBIC::API here.

And you can take a look at Data::DPath::Validator on CPAN here.

Or even the github repo here.

Of getting closer

So the work project has take a giant leap forward through burning some midnight oil last night. Through my failings, I've come to better understand Catalyst, how jQuery and it's plugins are nice, but not the right tool for the job, and how reverting back to trust Template::Toolkit, while ugly, gets the job done.

That said, I have a working datagrid. Some values are hardcoded at the moment to match the schema, but eventually, I will have it all automagical, down to the generation of the edit fields based on either data_type or additional metadata stuff into column_info.

So what was it that finally let me do the grid thing that I wanted? Ext. Now, I know some of you really disdain Ext as a company. And you have every right to dislike them. They commit the worst licensing FUD ever. Head on out to their site and see what I mean. Basically telling people that they must pay for a license if they are developing "commercial" software. Except that completely ignores the fact of how the GPL (version 3 specifically) works. in the grand scheme of things, I am developing "commercial" software in that it serves a business purpose and money is changing hands, but what Ext fails to make clear is that use of GPL software in those cases is covered. I am not distributing anything. I can eat up all of the GPL'd code I could ever want and never release a damn thing. The terms of the license are awesome like that. And yet it is "commericial."

Anyhow, beyond the bullshit with Ext licensing FUD, they have a rather solid UI model in their javascript library that very closely fits how desktop toolkits work. You can have windows, GTK/TK style layouts, decent events model, etc. Not only that, they have *gasp* a GridPanel. Throwing in some abstractions for "proxying" data, storing it, and providing a consistent interface for the UI to use, you have a very solid paradigm for doing web UI.

But there have been some bumps along the way. Catalyst::Controller::DBIC::API, while providing a fantastic set of functionality, leaves much of the customization up to the end developer to figure out. So I peaked inside the module and about went blind, heh. No, it really isn't that bad, it just needs some love and more of the steps it takes broken down some more into finer grained portions to allow the kind of customization that I required. For example, Ext's JsonReader and JsonWriter expect things in what I could consider a more normal format. Except DBIC::API doesn't allow you to make those changes. Oh sure you can override the format_list method, except you have no clue what to do without opening up the source. And there are lots of places like that. Where I had to explicitly dig into the source to figure out how to apply customizations. but in the end, I have the two talking to each other and even delivering metadata to the JsonReader/Store instead of hard coding things on the javascript side.

So after I get the work project squared away I will revisit DBIC::API and give it some lovin'. And then take that, with some Ext GPLed code, and build the real automagical datagrid that I wanted to build from the beginning.

Of failed attempts

I had really high hopes for getting Template::Declare to play nicely with MooseX::Declare (or even vanilla Moose), but that is proving a little too difficult. TD does a lot of symbol table fuckery. My big problem is that the template declarator is caller aware. It takes the caller package, and stores it into the dispatch table and also munges the symbol table for the current package to store the code ref. The problem is that if you want to subclass something and apply a role that advises some methods, that advice will never be called because the invocant was set in stone at the time the declarator was called. And since everything in TD is essentially singleton, class based, that means the place where your advice lives, in the sub class, is never called.

Oh sure you could use something like MooseX::CompileTime::Traits to advise various things, but the problem is that is very very global and not at all appropriate for what I am trying to do. You couldn't have two different grid things because the advice would apply to /all/ of them.

Aside from that, it has been a significant, constant uphill battle trying to make things work and I have run out of time. So I'll be throwing together the current work project using plain old TT, but still use DBIC::API and the javascript goodness that I found.

Consider me really put out by this.

Template::Declare + MooseX::Declare == Win

So, I've been playing around with getting TD to work with MXD nicely. This is the result for the most part:


What this will buy me is a more sane wrapper strategy with role application. I'd rather have 'around' keywords than the insanity that is TD's create_wrapper stuff. But mainly, I want type constraint checking that comes with MooseX::Method::Signatures. In the end, this is win. This lets me step back a little bit from TD, write some sanity, and then hook it up to the dispatcher via the template() function.

Granted it is a little unwieldy at first, but I am thinking I could distill this out into it's own thingy. We will see as the week progresses.

Datagrid

For those of you who have hacked in other languages, you know how easy it is at time to just display a grid of data (either to the user via the web or with a desktop application). Sadly, there isn't any as cohesive for Perl and Catalyst. And I really need one for the project I have at work. So this next week I am aiming to build one. The underlying tools will be Catalyst::Controller::DBIC::API, along with Template::Declare (specifically Catalyst::View::TD), using jquery plugins Collections and JSON-REST for the front end.

The premise is rather simple. Subclass one of the DBIC::API's concrete implementations (REST), provide an index end point that forwards to a TD view that spits out the grid. From then on, the javascript in the grid will actually manage the data via the REST api. To make customization easier, the grid, each "row" and each element in a row will have their own TD class. A helper script will be written that will generate these to subclass from the distributed classes to enable customization.

The goal is to provide four things: Controller that subclasses DBIC::API, Views that implement the grid and it's functionality, the ability to, within the Perl code, provide overridden behavior for all of the functionality of the grid (ie, methods on the TD classes that spit out the right javascript), and a helper class that makes it as painless as possible to expose a table via the datagrid in your application.

My hope is that with these four things in place, we'll have a coherent distribution with this functionality bundled (including the javascript) and people can stop reinventing this wheel and instead improve this one. We'll see how it goes.

Nerd Hackery

So I play a little space MMO that some of you may know (eve-online.com). And the game is rather vast when it comes to in game items. And a lot of these items have blueprints which allow you manufacture the items provided you have the necessary skills and materials. I was puttering around this past weekend trying to find a list of all the blueprints and what materials were required and simply could not find one that was current. But I did stumble upon a complete static dump of the game database for items, etc. And given some SQL-fu I could gather the information I wanted.

Sadly, my SQL-fu is not strong for scarily interrelated datasets, but that is why awesome people like mst came up with DBIx::Class and other modules of the same vain such as DBIx::Class::Schema::Loader. I got a hold of the database that CCP distributed and someone was kind enough to do a sqlite conversion. I threw Schema::Loader at it and got back a whole set Result classes ready for use.

Next step was to write a quick little script that would do the little calculations that I want and print out via STDOUT a CSV file suitable to be read in by OpenOffice. Earlier I read and commented on jnapiorkowski's blog about his very awesome MooseX::Role::BuildInstanceOfmodule. And so I took his module and used it in a class to spin up an attribute of the Schema class. Then I threw some MooseX::Getopt at the class to allow easy runtime option definition for things like what field separator to use.

Once that was all in place, the script ran and I now I have a rudimentary spreadsheet that allows me to find blueprints that take the materials that I readily have available. And what was my total time in developing this little 68 line script? 30 minutes.

Nerd hackery for the win!

More PCJ refactoring

So I spent the week analyzing in greater depth the patterns I've naturally seen as I started developing POEx modules and how they could be more readily applied to my effort to refactor and generally modernize POE::Component::Jabber. A couple of things came to light.

When I was working on Voltron over the summer to have ready for YAPC10, one of the more base components to the system was the ability to proxy sessions. Hence, POEx::ProxySession was born. And in there, I recognized that I needed that role that defined behavior for sending messages: POEx::ProxySession::MessageSender. And while looking at PCJ, I realized that I needed a more generic MessageSender role that could cover the obvious pattern of accessing a POE::Wheel and call put() with some data. Sending a message via a Wheel is such a common task, that I've decided to break it out into its own role: POEx::Role::MessageSender. And it covers just the behavior described above plus the ability to store context about a particular message.

In a case like PXPS, storing context is important since every message is really a request that is expecting a response. I modeled the protocol very similar to how XMPP functions with tagging a particular message request with an 'id'. And the more I thought about it, in an asynchronous system like POE, you would have to model most of your request/response interactions the same way. So that was built into POEx::Role::MessageSender.

And on a tangent while working this up, I debated how errors should be handled. I glanced at a bunch of different projects I had written in POE over the years and came to the conclusion that exception handling is rather lacking inside POE. POE, by default, executes each handler inside an eval. If for whatever reason that event dies, the $@ is captured and delivered to the owning session via a signal. If you don't have a signal handler installed for the "DIE" signal, then you actually won't get a chance to handle it. And you won't know about it "immediately." I put that in quotes for a reason.

The "DIE" signal is one of the "terminal" signals that POE checks for when dealing with signals. That means it will kill the session and propagate up, eventually stopping the Kernel, unless handled. But that can take a awhile if you have a lot of stuff going on, for some value of "awhile". This means if you are debugging your app while developing, you are going to be very annoyed at the all the stuff you have to wade through to trace the events that lead up to the exception. And this behavior is controllable via a folded constant, but I digress.

POEx::Role::MessageSender actually throws an exception if it is unable to obtain the wheel you want to use to send a message. This requires of course people to make use of the "DIE" signal facilities which a lot of people just don't do. So the second role that I am going to develop is POEx::Role::ExceptionDispatch. This will install a simple dispatch table as a DIE signal handler and expose a hash attribute with the keys being ClassNames and the values being tuples of session id/alias/ref, and event name. The reason I decided on tuples instead of postbacks or coderef, is I wanted to leave the door open for any kind of exception management, whether that be chained (install multiple handlers per exception class with each step given the option of stopping execution), execute-all (multiple handlers per exception class that all execute), or only one handler per exception type. Instead of building all of that logic into the role and trying to fit all of those needs, I opted to use the default moose method modifiers approach. If you want to do any complex exception handling, just advise your handlers with before/around/after. Problem solved. Code kept simple.

Just 'with' the role inside a POEx::Role::SessionInstantiation enabled object, and you have a much better exception handling interface.

POEx::Role::MessageSender is up on github(http://github.com/nperez/pxrms) , but it isn't finished at this point. I still need to flesh out the types and write tests. And soonish, I should have POEx::Role::ExceptionDispatch up there too.

My big picture goal is to develop as many of these kinds of roles as possible during the course of my work on various projects in the hopes that it will encourage further Moose + POE interaction. So any patterns of behavior that I see myself repeating between projects, I will be consolidating and generically developing into their own stand alone roles where needed. And I really hope more developers follow my lead by publishing roles to CPAN. I know many people write roles for their jobs, and with just a tiny bit of effort, for an enormous amount of gain, these generic behaviors can be made available. Because I see the future of CPAN in roles that can be easily composed and utilized

Refactoring POE::Component::Jabber

So tonight, I've started to refactor POE::Component::Jabber to make use of updated tools, including ones that I have written. That list includes everything from MooseX::Declare to POEx::Role::TCPClient. The following is a brain storming session.

The first step to is break the machine out of its mold, so to speak. Initially when I embarked on the 3.0 design, I recognized that there is nothing but an artificial limit imposed on multiple connections. Really, there should be no reason not to enable multiple connections. By using a level of indirection via POE::Component::PubSub, I could have delivered all of the pertinent state and connection information along with the received packet. But the changes introduced into 3.0 seemed a little overwhelming. I didn't want to fundamentally alter the behavior of one connection manager per connection. Now I want to change that. Having developed POEx::Role::TCPClient for the possibility of multiple outbound connections with no arbitrary limits, I solved the multiple connection problem. And in fact, consuming that role wipes out large swaths of wheel reinvention code.

And I think I could take that a step further and even allow a different connection type on each connection. Right now this is handled per instance of the connection manager with arguments passed to the constructor. The connection type determines which "dialect helper" gets instantiated and how the session is constructed. Currently, dialect helpers are a poorman's POE "role" in the sense that I hacked together an API for crudely introspecting which events the helper exposed. Then I used that method when creating the main PCJ session. The right way to do that is to instead is to have a /real/ Moose::Role represent the given functionality and consume it as needed.

Even then, that isn't enough. The dialect specific portions are only relevant during the connection negotiation phase. Once the connection is negotiated, those bits are never used again. And that leads me to think that perhaps I need to abstract that away and use a separate class for making connections. Then at the top level, PCJ merely delegates between a couple of different objects, mainly the connection maker, the pubsub component, and a controlling session for it all, but provides a uniform API for communicating over XMPP.

So we will see. I am going to work on it and see where it takes me.

Of Module Evaluation

One of the reasons I enjoy being involved in the Perl community is knowing that I can help people directly when it comes to evaluating my modules.

Recently, stephan48 in irc.perl.org/#poe needed to consider different options for doing some kind of multiprocess worker solution. A number of options were presented to him and I was lucky enough to garner his interest in trying out POEx::WorkerPool. It was a great learning experience in how people work through a distribution for evaluation.

The first thing he tried was to copy/paste my synopsis directly and run it. It had compile problems because I had made certain assumptions. For the most part, I wasn't expecting anyone to actually try to run it. In my defense, I consider the synopsis to just give a broad overview of the structure of use; not a full blown example useful for tweak-n-run solutions.

Second, he had no experience with the high level of tools on which POEx::WorkerPool is built. That meant that MooseX::Declare was foreign. The second stab he did at the code was attempting to shift $self from @_ which was giving him problems.

The third problem he encountered was the advanced and little used feature in POE that all exceptions in events are captured and propagated as a signal and as such requires a signal handler. Otherwise, the exception is devoured without so much as a peep to the developer. In my synopsis, I hadn't used all of the proper robust exception handling required to make sure things went smoothly. So in his attempt to simply have one worker with multiple queued jobs it failed because the method used immediately started the worker and prevented further job queue pushing. That failure was delivered as an exception. And to him, it was as if the magic wasn't happening at all. Only a single job would run.

All in all, we got him going eventually, and he came to the conclusion that perhaps this module wasn't the right tool for his job after all. And that's awesome that I was able to help him figure that out. I recognize that the tools that I build and use for my own projects may not be suitable for others. And while I see many advantages to using my solution for a fully scalable and maintainable system, sometimes, all you need is a little POE::Wheel::Run action to grease the wheels.

The upside is that I will be revisiting the documentation and perhaps build a mini-manual similar in structure to the Moose::Manual and the POE wiki. The casual developer has a problem to solve and doing 80% of the work in a well documented scaffold example really makes a difference.

MooseX::CompileTime::Traits

In developing POEx::WorkerPool, I did a lot of abstraction in order to make it easier to scavenge pieces of it for other uses, and also to allow customization on every level.

Initially, this meant that the classes inside POEx::WorkerPool are simply bare. They consume roles that contain the actual implementation details. This ultimately lets you consume those roles in other projects, gaining full functionality, without subclassing. The second feature of the bare classes is that they contained an import() method that did some magic to allow you to specify traits for those classes. Say you wanted to alter the behavior of how WorkerPool does its queing? You could simply write your own Moose::Role and pass it to 'use' to gain a global altering effect. Rockin.

But, it had its shortcomings. For one, I was doing all of the parsing logic, validation, and role application (with 'with' no less). This broke in several cases. Two, this was the second project to gain this ability, with POEx::Role::SessionInstantiation being the first. What I needed, was a proper encapsulation of this functionality.

And so MooseX::CompileTime::Traits was born. Now, some of you may be asking, why not MooseX::Traits::CompileTime? My ultimate reason for not doing that is that I didn't want people to confuse my module for a subclass of jrockway's module. MooseX::Traits applies traits at runtime using a custom constructor (new_with_traits()). So that means, your traits are actually on a per-instance basis. MooseX::CompileTime::Traits, on the otherhand, affects things at the class level. Globally.

I heard some grumblings that this might be a bad thing, but hear me out. While POEx::WorkerPool had multiple levels of abstraction applied to everything from the subprocess, to the worker in charge of it, to the pool itself, there was no clear way to tell it to do something different. To do that, I'd have to subclass up the chain of things inside POEx::WorkerPool to get the custom behavior I need in the lowest of levels. Perhaps that is a design issue, but the simple solution is to simply apply a trait at compile time without having to subclass a thing. I am not wanting this behavior to be selectively applied to some instances and not others. I want to change the behavior on all instances so it fits my needs without subclassing the entire project, basically.

And I needed to do this for work. This daemon that I am working on needed to do some initialization after the worker subprocess had forked. With MooseX::CompileTime::Traits though, it became easy to provide a role for the GutsLoader to advise some of the default behavior to do what I wanted to do. Now I can successfully invoke code after the fork has taken place without a large amount of work (the role to do this ended up being 10 lines).

So, if you want to give your classes (including the internals) the ability to absorb outside behaviors so that people can customize them while maintaining a very loose coupling, give MooseX::CompileTime::Traits a looksee.

Production Use

So I committed a mortal sin and have been reading Slashdot comments in the recent perl-5.11.0 announcement posted there. And like all conversations regarding Perl in the recent time frame, conversation inevitably drifts toward Perl 5 vs. Perl 6. And what I am finding in a lot of comments, not just on Slashdot, is this unhealthy apologist attitude in supporting Perl 6 by saying it is production ready, etc.

My beef is with this one comment: http://developers.slashdot.org/comments.pl?sid=1391409&cid=29627915

"Really? Can you give examples of problems in Rakudo that would stop it being used in production? Didn't think so."

I guess I should be writing this in a Slashdot comment, but I wanted to address this here and a little more broadly.

The number one rebuttal to the above comment is simply this: IO. I have previously attempted to do a naive port of POE to Rakudo and found it very lacking. There just isn't sufficient IO support in the current implementation of both Rakudo and Parrot to build any truly sound production systems. I can't remember a single system that I've implemented that didn't involve at one point some kind of complex IO such as multiplexing over multiple sockets.

Now I fully support the whole Perl 6 and Parrot thing. I really do. And when they finally get to the point where I can start building the tools that I currently take for granted, then we can start talking about production use.

The entire Parrot and Perl 6 movement has been nothing but awesome in terms of idea generation. The method argument syntax defined in MooseX::Method::Signatures, the concept of Roles, and all sorts of great things that have trickled back into Perl 5. And I even use those ideas in their current implementation for production system use. But the source of those ideas just aren't ready.

Of division of roles

The most recent release POEx::Role::SessionInstantiation now breaks out all of the various aspects of its inner workings into roles that are then composed into the master role. And now, via import, we can tell SessionInstantiation to compose other roles. It was a big refactor. But what did it buy? In terms of customization, it bought a lot.

The various segments were placed under POEx::Role::SessionInstantiation::Meta::Session::*. And so there is now Events, Implementation, Sugar, and Magic.

Events contains the various default events that POE would deliver to Sessions. Implementation takes care of the details of event dispatching and registration. Sugar provides methods that are delegated to the POE::Kernel singleton. And finally, Magic handles the tricky bits of making it all work with POE.

So now that everything is divided we can do some nifty things with the resulting roles. Let's say we wanted to build our objects without telling POE about them. All of that functionality is now stored inside Magic. It is simply a matter of advising the the BUILD method to not call session_alloc on the kernel instance. In fact, that functionality is now bundled up as POEx::Trait::DeferredRegistration.

And what if we wanted to do some initialization not specified in _start, but at that time from an external source? We can consume the Events role and advise _start as necessary. And this functionality is wrapped up in POEx::Trait::ExtraInitialization. It requires an arbitrary coderef at construction time that will be called as a method on the session at _start.

Now most of this was possible without the big refactor, but it would have been making use of under documented features inside SessionInstantiation. Now all of the bits and pieces have their docs and are segregated appropriately to allow for easier customization.

So what can you do with those kinds of traits? Aside from building better tools to manage resource sensitive operations such as loading modules before or after a fork, it will also eventually help with furthering POEx::ProxySession.

There is more to do on the Moose/Class::MOP front to finally make it possible to serialize in total a session, send it somewhere, reconstitute it, and activate it in the foreign POE kernel, but this is good step in that direction.

Murmur, DBus and Perl

Short post this week, but a neat project came up that I want to throw together this weekend.

Some quick background. Murmur is a VoIP server. Its typical application is use with the Mumble client to provide voice communications for computer games in the same vein as Ventrillo or Teamspeak. Together it provides a great set of features that include encryption, distinct channels, ACLs similar to IRC, and even user accounts. It is also useful for conference voice communication that provides superior voice quality than that of Skype or even conventional telephone conference calls that tend to be half-duplex.

So what does that have to do with DBus or Perl?

Well, Murmur's configuration all happens via DBus. DBus comes with a CLI app to sending arbitrary data along the bus properly addressed to a listening daemon. And just to add a user to Murmur the length of the command is unwieldy. In fact, you have to send two commands to set the password. You can do all sorts of configuration via DBus, but it is pretty hairy having to do it by hand.

So enter Perl. CPAN has a module called Net::DBus. So I am going to write a set of tools to administer Murmur and publish to CPAN under the App:: namespace.

Lots of good modules to use to make that happen including Moose, MooseX::Getopt, App::Cmd, etc.

By Sunday evening, I should hopefully have something that works (with stand alone libs that could be consumed via GUI app) and we can work on refinements from there.

You'll see the repo show up github soonish.

TryCatch + Throwable rocks

I hopped on the bandwagon a while ago since I started using these two modules in my own projects so I could have real live try/catch blocks including MyError->throw(). As a bonus I get constraint matching on which catch block to execute.

Lots of people have pushing the Try::Tiny that Yuval put together and that's cool too, but I like my constraint matching. I have had several cases where it makes sense to do different things based on the type of exception I have received. And, I do enjoy having the lexicals put into my scope too.

For me, since I am already using MooseX::Declare which uses MooseX::Method::Signatures which uses Parse::Method::Signatures, it is a natural extension that my catch constraints be defined with the same syntax.

Recently though, I ran into a bug in TryCatch regarding certain scenarios with returns inside the try block. I banged my head against the wall for a day or so to no avail until I manned up and asked Ash what was going on. And much to my delighted surprise, it was already a reported issue and already fixed in a dev release on CPAN. I pulled it down and ran my tests and magically everything started working.

Anyhow, what prompted all of this was my work on POEx::WorkerPool. In my first go around, I wrapped a lot of the potential error spots in try/catch blocks but some of the handling was inconsistent, so I wanted to go back and rework a lot of that to be even more robust.

Now when errors occur, signals are handled properly inside of POE, and are reported via PubSub. For the most part, catastrophic errors are mitigated and reported instead of taking the whole enchilada down into some horrible state.

As soon as Ash does a non-dev release of TryCatch, I'll do another release of POEx::WorkerPool that includes this new found robustness.

New Shiny

This past week I've been slammed; working to produce a new module: POEx::WorkerPool. Now, this will probably smack of NIH, but I looked at a couple of other good modules such as MooseX::Workers, and decided that I need to do my own that made use of the POEx tool chain and included MooseX::Declare.

To that end, the POEx::WorkerPool does the usual multi-process management and provides a mechanism to dole out jobs to workers, but where I feel it provides more bang for you buck is highlighted below.

-Easier Customization-
Each piece in the framework can be customized via traits defined in the 'use' statements of those pieces. Each piece does light coupling via Roles checking, so it doesn't hinder customization.

-Transparent Workers-
Workers publish a plethora of events via POEx::PubSub that allow for tracking of a job from the time the worker starts processing to even incremental progress updates

-Rudimentary Priority Modes-
Worker availability is effected by a two different modes: round_robin and fill_up. In round_robin mode, the next available worker is gathered in your typical round_robin fashion. And in fill_up mode, the same worker will continue to be gathered until it's work queue is full.

-Abstract Jobs-
The simple Job Role provides a light shell for whatever needs to be processed, but at the same time giving the framework some consistency for reporting progress, and identification.

And while those things are awesome, there is still room for improvement down the road. The API could be expanded a little further in the WorkerPool itself with regards to how the jobs get delivered. So there will be future releases as I work more on it.

Introducing new tools

It is exciting to get into a new project and inject new ideas and tools. The project that I am talking about is a generic report runner pulling its information from a database.

In its current incarnation it spins up from cron, and runs reports one at a time, but it now needs to run multiple reports including bulk reports. To do that I am making heavy use of POE, and a worker process pool. And, I am getting to use my own tools such as POEx::Role::SessionInstantiation, in addition to other great tools like MooseX::Declare.

Not only that, but I pulled in local::lib for dev purposes, and Dist::Zilla to handle the disitribution creation.

It's like being a kid in a toy store.

Of POE + Moose Awesomeness

So Florian finally dropped the bomb. Devel::Declare, MooseX::Method::Signatures, and MooseX::Declare were all released with all of the wholesome developed goodness that I wrote before YAPC::NA. So what does that mean? It means that I was able to finally release to CPAN a few modules.

POEx::Role::SessionInstantiation is the core of it all. It turns your Moose objects into POE::Sessions and manages the ugly for you.

POEx::Role::TCPClient builds on POEx::Role::SessionInstantiation and rolls up the common SocketFactory/ReadWrite wheels and gives you awesome functionality with minimal effort.

POEx::Role::TCPServer does the same thing as TCPClient, but in reverse.

I chose to stay the lower level for the roles because I found those pieces of the POE world ripe for Moosifying. The upper level Components are a bit unwieldy and I am not a big fan of them.

And once I work the kinks out of POEx::ProxySession, I'll be doing a release of that. I decided to add some more functionality with regards to getting notified from the POEx::ProxySession::Client when TCPClient fires socket errors and connect errors.

And then I'll be able to release an alpha of Voltron. Florian brought the parsed signature attribute on MooseX::Method::Signatures::Meta::Method up to public so that means I can rely on it now for doing signature comparisons instead of dumb string comparsons that I am doing now.

Expect to hear more soon on this front. I'm rather excited about getting to release my modules finally.

Working with Rakudo and Parrot

So after pmichaud had a call for applications for Rakudo, I came to a small epiphany that perhaps I should work on a naive port of POE to Rakudo. So this past weekend came and I started to do a little investigative work on what it would take to do POE on Rakudo. The first step is to get the core loop running. In Perl5 terms this means that select(2) needs to exist. In Perl5 world, select() is for the most part a C equivalent. There is another version of select() but that doesn't concern us.

Anyhow, select() let's us do synchronous multiplexing of IO across multiple filedescriptors. This is how POE does it's magic. As things come back from select() placed into the things-to-watch queue via POE::Kernel->select_read and select_write, those are placed into the Queue and get mingled in with all of the rest of the events that need to be dispatched. This is an obviously very simplified explanation.

So the first thing I did was go into various IRC channels to ask some very pertinent questions such as where is select() implemented? I dug around in the classes defined by Rakudo but could not find a definition of select. But someone pointed me in a couple of different directions and referred me to a couple of different people that were interested in the same topic.

Ultimately, it came down to this. I took a look-see into the Parrot source and found that, no, select was not implemented as a generic plain-old C-like call you could make. Instead, each of the Parrot socket objects have a method, poll, that will return to do it's status, if any. Internally, it uses select(2) to do this for the unixy source file anyhow.

Unfortunately, Rakudo has no parallel in the IO classes because it was decided to /remove/ select(2) (including the other version) from the spec. Apparently, people wanted to roll that up into the concurrency layer that doesn't really exist yet, because it is pretty instrumental in event loops. In the mean time, I'm for the most part stuck. Sure I could do a IO::Select port to Rakudo (and had planned to do just that), but I would have to implement it using the non-portable Q:PIR construct and call down to the Parrot sockets methods. I'd have to learn PIR, ontop of any augmentation to Perl6 that I barely know. Plus, it would only work on the Rakudo implementation due to the use of inline PIR.

With that said, I decided to stop looking at doing a naive port until the IO layer is fleshed out a bit more and I have safer abstractions from which I can do simple things like use a select-like idiom that is extremely common for writing socket based network servers. The Parrot and the Rakudo projects have come a very long way in their years of development, but I just don't have the time to be writing core infrastructure pieces and also maintain and further develop my other modules. I really understand the catch-22 that Rakudo and Perl6 in general have when it comes to specs vs implementations, but I can't really do much to fix it without investing a lot of time and effort coming up to speed on the thoughts behind a lot of the decisions (such as removing select()).

I tried to approach this project from the perspective of bringing current Perl5 tools forward to the Perl6 world. That means attempting a naive, get it running, port that doesn't bother with any of the new fancy idioms. At most, I wanted to adjust the sigils, and do some light source filtering by hand to cover the large breakages. Document what I found for other projects looking to do the same. And go from there. I couldn't really get to square one because of the specification vs. implementation situation.

So while I certainly sympathize with these projects, I just don't feel they are ready yet for the casual developer (eg. me) to even take the first steps. I would like to at least revisit the issue for the Rakudo Star release, however, given more of the IO layer has been implemented and I can do what I want without resorting to platform specific inline assembly.

Let the flames begin :-/

Not enough time

"It would have been really cool to use module X because I could have used the same config for both things. Just didn't have enough time." Or "there just isn't any time to go back and redo some of it." Or "this thing is hacked together because we didn't have any time". And that is kind of hard to wrap my head around. Yes, time is always against us when the customer is yelling for feature X, complaining about bug Y and deadline Z is looming. They are writing the checks, after all. But, there has to be a point where something needs to be more "right" than "right now." I've worked on projects with some tight time lines. I've had to compromise on features, workaround bugs, and ship something I felt wasn't 100%, but I don't think I've kludged something together without ever revisiting it and correcting it.

As a programmer who takes pride in his work, it is hard to hear these kinds of things. Because to me, it sounds like the people making those statements are resigning to accept that the hackery and the shortcuts are worth more than the increase in maintenance time down road.

So how does all of this apply to Perl? Backticks. Network daemons that don't use POE. Manual parameter validation. Etc. All of the things that make Perl a swiss army chainsaw, also make it very easy to forget to engineer your software. That is why I like things like POE, Moose, Catalyst and others. Those modules help you to engineer your software. They discourage you from doing things like using backticks or reinventing wheels.

And using those tools to begin with decreases your development time. Those projects have solved a lot of the hard problems for you. They are forward thinking and keep their userbases in mind.

I wish I had a solid answer for people with the perpetual time problem.

One Step Closer

Florian has taken the time to look at my work on MooseX::Method::Signatures and committed method trait changes to his master. I am very excited about this for two reasons.

One, it really is much simpler than my implementation. No need to parse Perl with PPI via Parse::Constructor::Arguments. I am not sure why I didn't consider his approach in the first place, honestly. It makes much more sense. I am going to blame Cory because I picked up where he left off, heh. No, really, I should have taken a step back and realized I could do what I wanted to do without invoking PPI.

Parse::Constructor::Arguments is going to stick around even if its use has been superceeded by simpler code. Someone else might find it useful. I still need to test it against the latest PPI to make sure it doesn't break though.

And two, once integration into MooseX::Declare happens, I can finally release all of my modules I wrote for YAPC::NA. I've sort of stopped work on Voltron and on POEx::ProxySession largely because I am waiting for Florian to catch up. I am excited to start back up again and finish fleshing out the automagical reconnect logic in ProxySession. From there, Voltron is going to get proper signature matching support via Parse::Method::Signatures.

In other news, I think I've nailed down my learning language syntax and semantics. I've covered a core language and then an object oriented extension. Next up is manipulating flex to get it to lex it.

And finally, yes, I did get the job. I start this coming Monday. Still keeping my name in the hat though for some other things. This job is only a contract gig for six months. It might extend, but then again, it might not, so I might as well keep my ear to the ground.

Preparing for a Perl Interview

So I have an interview tomorrow with a prospective employer. The job is supposedly a Perl Software Engineer position, but the skill set they want is all over the place. So we will see, really, what they want.

But I like to prepare as much as possible for interviews. And it depends on the interviewer what kind of questions they will ask, but I like to cover my bases by reading over the existing core documentation. That includes all of the obscure things inside perlvar, and even the deltas to know just when I can't use some modern feature. It seems like every interviewer has their "gotcha" question: the one question they weigh the heaviest in their assessment of your skills. Even I have them.

The Perl knowledge space is rather large when you consider what is packed into the core language. Everything from complex data structures, special global variables, an entire compact, and somewhat hard to parse-by-person, sub language (regular expressions), context depending expressions, and so much more is all crammed into Perl. There are bits that I am very much not intimate with: perlguts or XS in general, formats, pack/unpack, and I am sure there are others. Generally when I need to venture into those parts of the language, I will grok the concepts for the duration of my need and then it is back into the ether with those. But sometimes, those hard parts end up as someone's pet interview question. "Write me a format that produces this output." "Here is a specification for a message encoded as binary, write me a pack/unpack string that produces/consumes it." "We have a simple C library that we want to use in Perl, write some XS that encapsulates this functions usage."

And then sometimes, I am handed a dry erase marker and told to start writing code. That is very hard. Not that I consider a proper text editor with syntax highlighting and brace matching a crutch, but it is a rather limiting experience to write on a whiteboard. In front of people. People that are critical.

And even though I am critical of the tests that are sometimes given, I prefer them over a live whiteboard session. I guess part of my insecurity is that I have not done a lot of pair programming so I am not used to idea of someone looking over my shoulder constantly with a critical eye toward what I am doing.

How tomorrow is going to go, I am not sure. I am nervous. I am brushing up on the esoterics just in case. I just hope these guys have taken the time to go over my public work and save themselves some effort when it comes to asking the silly questions such as "What keyword would I use to sort an array?"

Foreign Country Hacking

So I've been in Mexico for the past week visiting family, and I haven't had the luxury of time to sit down and continue to work on any of my hacking projects. I did bring some materials including the docs for flex and bison to work a bit on my toy language, but haven't touch them.

When I get back there are a number of things I need to look into (besides trying to find a job) in no particular order:

1. Get the lexer grammar written for NULL (Nick's Useless Learning Language)
2. Rewrite the INSTALL doc for perl core (make it exhaustive of all the options and then list better examples on types of installs).
3a. Bug Florian to merge my code
3b. Release all of my modules from YAPC
4. Put my recent work on the POE wiki and work toward more POE+Moose interaction

I am heartened to see that RGS cherry-picked my patch and committed. It makes me want to do more work in that area until I get my feet wet enough to contribute meaningfully with code.

Light post this week. I have been too inundated with international travel to do much more than eat, sleep, and fiesta :)

Programming tests with Perl

I figured since my last post was so provocative, I've decided to write about something completely uninteresting and uncontroversial. Who here has had to take a programming test as a step in the interview processes? I know I have a few times.

Java tests. C# tests. Etc. Usually they want you to write from scratch some solution to a very contrived problem. And that is fine. They want to see a sample of your work and how to approach your problems. Sure as hell beats standing in front of a group of critical people with a dry erase marker in your hand and a massive expanse of whiteboard behind you.

Typically when taking a test for another language, writing stuff from scratch is mostly the norm. Oh sure there are open source libs out there for various languages, but when it comes to Perl and the massive CPAN, those other libs are dwarfed by comparison. So when I come across a Perl test that wants you to write something from scratch when a myriad of distributions on CPAN solve that exact problem, I get a little worried. Does this corp environment discourage use of CPAN to avoid reinventing the wheel? Does NIH permeate their corporate culture to the point of detriment?

In the past, I simply fired up cpan, changed a few config options and made sure things were installed into a local directory. The proctors of the test were a little intrigued, but mostly they were against the idea. They wanted to see me waste my time writing the wheel again. Ultimately, in that particular instance, I didn't get the job. And probably a good thing, too.

Of all the tests I've taken in the past though, they given me a machine, most of the tools I would expected to use, and a fast enough internet connection to use for searching for hints. And this makes sense. You wouldn't ask a graphic designer to create a design without some graphics program like Illustrator or Photoshop or tell an architech to draw up some blue prints without advanced CAD software. So why would a software programmer be any different?

Recently, I took another Perl test from a recently well known municipal auction provider, but this time I wasn't supposed to use a proper editor with syntax highlighting, I wasn't supposed to attempt to compile my answers, nor was I supposed to use any internet based resources. My face wrinkled up into the weirdest frown. They wanted to duplicate a "written" test environment. How does that have any bearing on the job they are potentially asking me to do? Not that they could enforce that since it was just a dumb form on a page without any javascript (I certainly didn't turn it on to see).

So I used the tools at my disposal to solve their contrived problems to the best of my ability using the tools that I use to develop software: vim, perl5.10.0, etc. I also made sure that my solutions were wrapped up into proper classes (via MooseX::Declare), reusable, and took advantage of modern Perl idioms. Save for some hairy multiple sorting that was giving me an issue, all of my answers compiled the first time without syntax problems and all ran as designated in the problem without issue. I tried to be as thorough as possible without going overboard. MooseX::Method::Signatures via MooseX::Declare saved me a lot of time from validating inputs, etc by hand, so that was a big win. If I had had to do that myself, I wouldn't have finished the test.

For some reason, I didn't get this job either. I asked for a little more feedback but I don't believe it will be forthcoming, which is their prerogative. It would be nice to know how I did on their test.

But considering my short chat with one of their developers at YAPC a couple of weeks ago revolved around "Why would you use POE when we could just spin up another persistent mod_perl instance?" perhaps it is good that place decided to look elsewhere.

So just for the record, if any of you future employers out there want to give me a test, please remember that I am going to use all of the tools at my disposal and give you an honest effort at solving your problems. If you tie my hands behind my back and then are disappointed with the result of my painting a bikeshed with my feet, that's your problem and not mine.

DarkPAN SchmarkPAN -- STOP THE MEME

"The DarkPAN is too big and scary to change Perl 5's defaults all even by the time of Perl 5.14" -- chromatic

DarkPAN was discussed quite a bit at YAPC::NA. But really, what is it? How is it defined? Who are these people that have a vested interested in Perl and yet do not participate? I think attempting to appease a faceless entity with no defined boundaries has fail written all over it. It is fear mongering.

If we break it, they can choose not to upgrade. What about bug fixes?, I hear you say. That is the joy of open source. There are lots of unemployed hackers out there that would be willing to backport those fixes for you. Oh heavens forbid! Actually paying open source hackers to write code! The sky is falling!

We need to stop this meme of DarkPAN. Perl does not belong to DarkPAN. It belongs to us who participate in its wellbeing. And if the amorphous corporate overlords are angered by our changes, then they need to get involved and list their grievances, just like Joe OpenSource Hacker.

But until we hear /anything/ from this unholy DarkPAN, I say we make the decisions best for our language of choice. I once had a professor that was fond of saying "Silence is tacit consent." And in this case, I would say that is extremely apropos of Perl stewardship.

Half way through

Oops! Missed my blog post for yesterday, but that is only because there has been so much awesome stuff going on here in Pittsburgh, PA at YAPC::NA::2009.

I gave my POE::Filters talk today and was a little stressed about it. I didn't finish the slides until last night, and then I guess I talked a little too fast. But it was alright in the end since no one really knew what the hell I was talking about except the POE hackers in the audience.

I gave (and will give another tomorrow) POE primer at the front of my talk which apparently was derided as not catchy enough. Shrug. It was either that, or have a bunch of people stare dumbfounded at my code examples, wondering what was happening. Sadly, I was scheduled against such heavyweights as nothingmuch and jrockway and so I had all of 20 people show up.

On the plus side, lots of hallway track time has lead to some great conversations and involvement in some good things such as hacking on perl core.

Hacking on perl core. That is pretty deep stuff. My C isn't exactly super strong, but it is really important I believe. Important enough that I want to get involved. There is a serious problem in that the core of perl is not documented. I even made mention of that for the EPO's document bounties. It got votes, just not enough to offer a bounty for it.

So I am getting involved. Our first task is to consider the tests between miniperl and perl and see where the short comings are for miniperl. A deep dive into miniperl gives us a good look into things like the tokenizer, etc.

And write about it. Blogging about it exposes the tribal knowledge to the intertubes.

And so a new line of topics begins!

Getting Closer

With YAPC::NA only a handful of days away, it is crunch time for me. I had set a deadline of last night to have Voltron finished, and as typical, life got in the way. I am close though and should finish today. But the schedule I've made for myself for this week includes a 'few days' to make slides and solidify my presentations. So I have a little buffer room, but really, I don't want to be scrambling to do slides the weekend before or, gasp, during someone else's talk.

When it comes to building a complete stack for a project one of the things I love most is recognizing a missing functionality and implementing it in the proper layer. This time around, it had to do with arguments passed to callback events from TCPClient. In single connection setup, it was silly to consider context for your callbacks. You only had one connection, so you know you were operating in a particular context. But at the same time, there was nothing in the TCPClient implementation that limited things to a single connection. When I got up to the ProxySession layer, those similar behaviors followed. While there were no limitations in the code to only one connection, the API gave you no way to determine from which connection a callback was originated. And I was okay with that for the time being until I got up to the Voltron layer and realized, it was a silly limitation and imagined a single application connecting to multiple application servers. So down we went.

TCPClient was changed to allow better per connection tracking, multiple simultaneous connect calls, and some other details. Then ProxySession was changed to always return the connection id in callback events, and also enable sending along user data that would show up in the callback. Another oversight was my decorator roles. ProxyEvent was it's own thing when really it should compose Event because you can't have a proxied event without a regular event. That cleaned up some ugly syntax in my tests.

And while all of this is great, and I am making lot of progress, Voltron still isn't complete because I've been adding functionality into my lower layers. In the end though, it will make Voltron an even simpler module with a lot of the complexity encapsulated away.

After I finish Voltron today, I have to start on a simple XMLRPC server/client implementation for my POE::Filters talk. And while I want to Moose that up a bit, I will likely do it in a very non moose fashion (save for POE::Filter::XML::RPC since that was my first foray into the Moose world. Attributes rock). The again, I really really like using POEx::Role::SessionInstantiation. It makes my code so much cleaner and all of the POE bits less obvious. We'll see.

Next week, I'll be blogging from YAPC::NA and will likely see a lot of you regular Perl Iron Man readers/writers there.

POEx::ProxySession is alive

So I have been putting time into my IKC-like replacement and with last nights (err, this mornings) commits, I have a working server/client setup that successfully proxies a POEx::Role::Instantiation composed class.

So like I was explaining a couple of weeks ago, I needed a simpler IKC that provided persistent local subscribed sessions that proxied posts through the server, back to the original published session. Mainly, I need Voltron to have access to these proxied sessions so that it is trivial to setup applications and to post events to them using PubSub.

Anyhow, it was an arduous task getting this thing work. There were some problems, naturally. The biggest problem was in the client. Creating a proxy session using the serialized Moose::Meta::Class was rife with issues, largely because Moose goes to great lengths to optimize it's meta internals via coderefs. And Storable+B::Deparse doesn't seem to work as advertised, especially when there are hooks introduced into the parsing process via Devel::Declare. So I couldn't count on coderefs.

So that meant after getting the deserialized Meta::Class I couldn't touch any of the methods on the object itself. But as long as I broke encapsulation on the Meta::Class (mainly via ->{methods}), I got a hold of all the meta data that I needed.

The second issue had to do with parsing method signatures and in the end it made a lot of sense once I understood what was all going on in the environment. MXD makes use of namespace::autoclean which scrubs the symbol table post intial compilation. That means that all of my imported POEx::Types were getting scrubbed away, which was a problem when I was passing method signature strings to MXMS::Meta::Method->wrap(). It couldn't find the symbols. And I chased this behavior for a while. Traced into several modules, and eventually wrote a simpler test case that would give the same result. And yes, it had to do with symbols. So I appened the "is dirty" option to the Client class and like magic things started working.

But in my symbol exploration, I discovered that POEx::Role::SessionInstantiation in during its anonymous class cloning /all/ of the symbols not native to the class were lost, which is a big problem. So that got fixed.

Beyond those problems in making several different technologies work together, the message data structure went through a couple of revisions to the point where a message identifier is now required, and result/delivery semantics correctly work. In addition, the message data structure itself is lightly checked for correctness via the subtype where clauses which means validation happens everywhere. It's pretty rockin. And, a new Role is provided to decorate methods to indicate which methods should be proxied on the subscriber side(POEx::Role::ProxyEvent).

And the one last thing that was missing was a shutdown event for POEx::Role::TCPServer and TCPClient. In the tests for those modules, I was manually clearing out the stored sockets and aliases, etc, but that wouldn't really work for the tests for PXPS. So a quick implementation and update to tests, and like magic it all works.

Unfortunately, it looks like I will be going to YAPC::NA with a vast majority of the dependencies for Voltron living only in git repos rather than on the CPAN. Which isn't so bad, but it severely limits participation from the audience when they have to basically duplicate my environment just to play with it. And it isn't an easy environment to setup considering you have to clone a number of repos, AND have Dist::Zilla installed.

For the conference, I am considering at least bundling everything up in an installation ready state and hosting the tarball. Wouldn't be a lot of work, but it adds on to everything else I still need to do. Two weeks until the conference and Voltron needs a lot of work. And, my POE::Filters talk needs a simple implementation still. I'm freaking out a little bit. Long nights/days ahead of me the next couple of weeks.

Closer to Magic

This past week was a light week development wise. Mostly, I am blocking waiting for my dependencies for Voltron to catch up and merge in my branches. One of the blockers was method trait arguments in MooseX::Method::Signatures. I am not really a firm believer in passing arguments to these things since it is compile time (in terms of MXMS/MXD), so you are stuck with static data. But it was deemed important enough to require implementation. So I worked with Cory Watson to figure where his implementation was falling short.

Initially, the first attempt at traits is what I merged into my local repo and made it work. It was kind of a naive approach where the stuff directly returned from the Devel::Declare context was just rolled up and applied. And the arguments were discarded. It works. The problem with the arguments is that you get them as a string back from the context. And how do you turn a string into a data structure suitable for constructor consumption?

You could eval, or you could parse. I chose parse. So I spent the weeknd and part of the week implementing a PPI based module for parsing Moose-style constructor arguments (eg. Foo->new(attribute1 => ['foo'], ...) ). And I am happy to report that Parse::Constructor::Arguments parses arbitrarily nested static declarations for arguments and returns a hash ref suitable for %{ $hashref } in a constructor.

In the mean time, Cory had taken another stab at doing traits+args. And this work really had promise, but he was stuck. And an hour of delving into it, I realized a couple of things. First, he was overwriting the caller package initialized $meta which means that the method, once properly instantiated, was not applied appropriately. And secondly, his arguments weren't being applied at all, not sure what to do with what his shadowed method was getting.

The first was easy enough to fix, just add in another lexical. The second was a case of applying my previous PPI work and also Hash::Merge to roll up all of the arguments from all of the traits so they can be all be supplied to the new_object method once.

And after a little clean up and commit amending, all tests were passing and based on the lastest Florian master.

Now I just need to hound Florian until he either merges in the branch or implements his own solution for trait arguments.

Of the long road to usable POEx::Roles

This past week was punctuated by lots of frustration. But the good kind really. As of yesterday, I have a working POEx::Role::TCPServer. That is, the simple POE::Wheel::SocketFactory and POE::Wheel::ReadWrite bundle on top of POEx::Role::SessionInstantiation(pxrsi) works as expected without surprises. It took all week to get there really.

Pxrsi had problems. A lot of problems. While in the simple cases, it would work as designed, as soon as you threw in a Wheel that fiddles with the states of the instantiating session, it go sideways, off the cliff, into the valley of lava, inhabited by flesh eating zombies. The biggest problem was that my advised methods like _start were getting called in an infinite loop, especially if the first _start executed touch the states and kicked off a _clone_self. Chatting with Matt Trout about the problem had a startling revelation: when constructing an anonymous class, roles that you give to the method are applied. Which makes sense if you if you step away from my crack filled idea of what I was doing (anonymous class clones to enable per instance method level changes). So my methods were getting advised multiple times.

And so that led to lots of hilarity. in TCPServer, I am instantiating a SocketFactory on _start. Well if there are multple _starts, the second will fail because the first has already bound to the port. But of course, that isn't obvious at first. In the depths of the debugger, I was noticing that as roles were being applied that the instantiated SocketFactory was /going out of scope/. WTF? So yeah, I chased that one for a little while.

So I made a few changes to pxrsi: only clone once when needed, revert tracing from fancy advised methods to simple statement before invocation, etc. The only clone once thing made a lot of sense. It would only happen if an instance received a _register_state event, and from that point on, you are working with an anonymous clone class specific to that instance anyhow. So that in the end is a big performance gain not having to clone the class every time a state is registered or removed.

Awesome, but then I came across another problem. I was making use of Moosex::AttributeHelpers in POEx::Role::TCPServer for wheels management and I had defined a number of "provides" methods. Of course, the _invoke_state heuristic for determining events from non-events was failing. And that is the problem with implicitness, in this case.

But the day was saved when Cory Watson announced work to support method traits. Holy crap that is exactly what I needed. MooseX::POE (mxp) had added a declarative keyword for defining events. And it made sense because of how mxp was architected around POE::Session. I wanted something a little more magical. So it wasn't a day or so before Cory had the method traits stuff working. A couple of git remote updates later, and I had declarative events. So _invoke_state and _register_state were rewritten to handle checking for a POEx::Role::Event role and like magic, the simple cases were working.

So I scurried back to POEx::Role::TCP and it was still failing. The coderefs installed by the various wheels shipped with POE obviously expect a POE::Session API compatibility. Hrm. Then reworked part of the method responsible for wrapping coderefs delivered from wheels (a horrible hack, I might add).

Then mysteriously another failure cropped up in pxrsi that took me awhile to figure out. My test wouldn't even run. Of course, this was after a week of changes from Devel::Declare, to MooseX::Method::Signatures, to MooseX::Declare (basically the whole chain of dependencies under pxrsi). The message I was getting was rather cryptic. It (Moose) was complaining that it couldn't find one of the advised methods in the inheritance hierarchy for the Role. It kind of blew my mind. It was working before! But then I remember that I had made a small change to the test to advise one of the methods in the class being defined rather than in a class defined from an event handler.

The way MooseX::Declare works with roles definition is that it defers application of those roles until after the class has finished parsing. And with my changes, if you remember, method modifiers don't get executed until after roles have been applied. So I was a little perplexed. I don't know if this is a bug in Class::MOP/Moose or what, but the roles being applied weren't actually loaded, yet. But once you had an instance of the class, your role was ready to go. So I patched MooseX::Declare to call load_class as the roles were encountered to ensure they are loaded before being applied. And that fixed my advising.

So what is the result of all of this? POEx::Role::TCPServer works and is very simple. Scarily simple. And you can advise any portion of it. This means that you could around advise handle_on_connect to SSLify a socket before letting TCPServer do the mundane stuff, etc. It is pretty cool.

But ultimately, why am I reinventing the wheel, you may ask. I came across a tiny little problem when working on Voltron last weekend. I need a much, much simpler model for session proxying than what IKC provides. Basically, I want to be able to "subscribe" to a remote session and have it instantiate a local, persistent proxy session. So I need to write that. And well, if I am writing things like this, I might as well eat my own dog food and build up all of the toolchain to make that happen. So that includes things like TCP server and client roles, etc. It worries me somewhat because I only have a month left before YAPC and that means, I need to get this thing working as quickly as possible. Hell of a time to decide that I need to replace one of the core components.

I really feel that writing these roles will decrease complexity in developing POE applications. For me, it always felt like there was this tension in writing good, clean OO code and making use of POE. Oh sure you have object states and whatever, but the machinations of POE would shine through for every constructor that instantiated a POE::Session object. I want to make the encapsulation complete, eventually. Ricardo Signes is working on some MRO::Magic module that promises some awesomeness. It would be very cool if all your method invocations, magically, behind the scenes carried out the POE stuff without having to be too concerned about it. So posting, yielding, and calling would look like ordinary method invocations on an object, but the object itself would essentially be a proxy. Still a long way off from making that happen, but we're getting there.

MooseX::Declare and MooseX::Method::Signatures

In my on going efforts to have code written before YAPC::NA for my Voltron talk, I came across an odd problem. Well, first, let's back up and talk a little about Voltron. Basically, I am combiningPOE, IKC, PubSub and Moose to create an application server that can introspect into connected pieces and automagically setup applications and participants to each other. The introspection is the key part.

Anyhow, MooseX::Declare supports a method syntax that is pretty cool. Basically it gives you a Perl 6 feel in Perl 5 land. That said, it was missing an important feature which is return type constraints which I had found out experimentally. I had made a naive assumption that the method syntax had feature parity with MooseX::Method::Signatures. It didn't.

So I make mention of this in #devel-declare to Florian Ragwitz, Robert Sedlacek, and the others. Come to find out MXD /reimplemented/ bits from MXMS, instead of making use of the module directly. So I peek inside the code for both modules. Ultimately, both modules make use of Devel::Declare to work their magic, so that helps. But they way they go about using that module is slightly different.

MXD actually does most of it's processing via a Devel::Declare::Context::Simple object. The context is contained, and lots of stuff is delegated to it. If I was going to make MXMS parse methods inside of MXD, I was going to need to pass the current context to MXMS and have it operate on that.

MXMS doesn't instantiate a Context::Simple object, it is the object. Or rather a subclass once removed. So instead of delegating operations to a separate object, it is making calls to $self. Obviously that isn't going to fly.

First, MooseX::Method::Signatures needed to have it's context split out from the rest of the code. MXMS was actually subclassing Devel::Declare::MethodInstaller::Simple which is a subclass of Devel::Declare::Context::Simple and it was doing that to get a method for parsing attributes (not the Moose kind, the Perl 5 kind). Handy, but if I was to split out the context and absorb a context from MXD, it needed to be Context::Simple and nothing more. So I copy/pasted the attribute parsing code in to MXMS and tweaked it to operate on a separate context. The rest of the module needed all of the instances of $self->foo turned into $ctx = $self->context; $ctx->foo. But once that was done, it was easy street from then on out.

Then for MXD to make use of MXMS, all I had to do was instantiate it, pass it a context, and tell it to do it's thing. Rockin'.

Except now, my previous work to defer method modifiers until after roles consumption broke. MXMS had no clue that it was supposed to defer method modifier application. Nor should it. So how is that reconciled? Well, what if MXMS had a coderef to call upon method application, so that you could override the default $meta->add_method or $meta->add_method_modifier? So that is what I implemented. Now in Keyword::MethodModifier, I simply pass in a closure around what should happen (pushing method modifer application coderefs onto the modifiers_store for later execution).

Awesome. Then Ash noted that applying the around method modifier to something that provided an invocant would break things because you would end up with /two/ invocants (the $orig coderef and the original invocant) and that would break Parse::Method::Signature. And because the old MXD code was playing with the method signature and creating a Meta::Method object itself I had cargo culted that behavior since I didn't want to fundamentally change anything. So my shiny was broken. Hrm.

So we discuss options on how to fix. I suggest that perhaps we make modifications to Parse::Method::Signature::Sig to allow for creating and manipulating them, because that is what I was doing anyhow, but with raw strings, so I could change the invocant and futz with the arguments. Then Ash provided a Good Enough solution which was to dig into Meta::Method inside MXMS, and inject the $orig parameter. And so I did. Anytime the declarator was 'around', I injected $orig both into the lexicals and into the positional args. Clean fix.

So after all that what is the next result? Full support of MXMS style methods inside of MXD including return value type constraints. So now in Voltron, I have full introspection ability which will be need for autowiring participants to applications.

Naming things is hard

So this past week, I finished up the evil arcana known as POEx::Role::SessionInstantiation. Well, at least that is how I ultimately named it. But that decision to name it as such wasn't an easy one to make, largely because I was straddling two separate fiefdoms and neither could seem to agree.

Originally, I had thought of naming the module POE::Session::Moose. At the time, it made sense. It was a Session implemented in Moose, except it was more than that. And it wasn't a subclass of Session per se. So I queried around for possible name ideas with somewhat neutral persons. Fresh on everyone's mind was the idea of using POEx as a root namespace. Matt Cashner actually pushed the idea to the forefront recently with a /topic change in #poe. And since my module was really a Moose::Role, POEx::Role seemed like a great starting point.

Then we kicked around ideas for the last name. Matt Trout suggested readability within the Moose system (ie. ->does(RoleName)) which tickled all the right language processors the right way. Sessionify? Sessionize? Kind of hard to pronounce. SessionInstantiation came up. ->does(SessionInstantiation) reads really well. So it was settled: POEx::Role::SessionInstantiation. Even shortens to something goofy: pxrsi or PEH-ZERCK-SEE. But to make sure I wasn't going to be stepping on anyone's toes by leaping into the POEx namespace first, I broached the topic on #poe.

It was not well received. Many reasons were given, but the biggest reason was control. The POEx namespace from previous conversations on the matter years previous, was supposed to be a reboot and salvation from the POE::Component "swamp." Where the POEx namespace and its descendents essentially define an interlocking, guaranteed behavior. A laudable goal. One that I thought my carefully considered name aligned with. There are many aspects in the POE world that could easily be reimplemented as Roles. As we know from chromatic's evangelizing, Roles == behaviors.

My efforts to get the ball rolling on the conversation, to get a decision made, to get some sort of boundaries drawn around the problem (at least defining the various aspects of the POE world that would be better represented inside a POEx::* namespace) were futile. And finally it was decided that perhaps it would better fit in a Moose namespace (such as MooseX).

Hrm. Not discourage, I started the conversation about naming in #moose. I was met with resistance. "This seems more POE specific than Moose specific." And in the near past, the discussion on having a MooseX::Role namespace was just line noise. So there was no where for me to go. It was offered to perhaps nestle inside MooseX::POE, but that didn't feel right largely because Chris Prather's approach and mine are so completely different.

Eventually, I engage both camps at the same time and let them duke it out. Back and forth on naming happened more or less like a tennis match with each side reasoning against it being in their space. Talk about feeling the love, heh. More suggestions came up to put it in the Class::* namespace, but that doesn't work because it is a Role. Someone thought maybe start a new Top Level Name: Role, but that doesn't suggest Moosiness. But then again, Moose is the only object system that implements Roles in p5. All in all there was a lot of mental masturbation.

And in the end, what was decided? Nothing. Consensus was never reached. So after a few days of that, my new, shiny code was still sitting in a git repo without CPAN distribution. So I was left to decide on my own really. The way that the CPAN indexer works, there are no real constraints on names so long as there isn't a collision. My efforts were mostly to not piss in people's cheerios. And that didn't pan out. Decision by committee simply doesn't occur with any kind of efficiency or expediency. Especially when that committee is made up of two differing groups each with their own ideas and designs on naming things.

POEx::Role::SessionInstantiation stuck. I liked it. It was generated with outside opinions before engaging each camp. It reads nicely in practice. It properly describes the behavior bestowed upon the composed class. And so it was uploaded to the CPAN.

The only good, I feel, that came of this was an absolute refutation of the concept that Perl is a dying language. There are a plethora of cabals out there, each with their own fiefdom in the Perl kindgom. And that is healthy. There are so many expansions of technology within the Perl space, that while we may all not agree on certain things (names for example), we can all agree that our language of choice rocks.

POE + Moose with POEx::Role::SessionInstantiation | You, too, can do evil dark magic with Perl | IRC is a better friend than Google

And deeper down the rabbit hole we go.

So it all started in IRC the other day this past week, with Martijn van Beers talking about using MooseX::POE to rebuild Client::TCP. And I had a kind of epiphany. What if POE::Session were actually a Moose::Role. Think about it. What if you could take any object and turn it into a Session with out invoking the arcana yourself? That would be pretty cool right? Suddenly, the object_states of POE::Session become a first class concept. Subclassing a Session because you just want to make a couple of tweaks to a couple of methods becomes as simple of providing around advice for those methods. Your object becomes the ultimate POE cyborg. But how could a Role be so transmutative. With lots of fuckin' poking and prodding anything is possible.

So I shared this little epiphany and started hacking on it that night. Until dawn. And by then, I had a very simple working solution. After consulting some great POE::Session implementations on the CPAN such as POE::Session::Cascading and POE::Session::PlainCall (and stealing their ideas), I figured out that I needed to call session_alloc to register with the Kernel, and also to implement _invoke_state to receive events. So I did.

Inside the Role, I provide a dummy BUILD sub and then after advice for said BUILD sub (in case the consuming class has a BUILD and overrides my dummy one) to register with the Kernel. That took a little bit of brain power, but was oddly enough provided by Martijn again by regurgitating that exact information I needed from the #moose channel about this very topic. Pure awesomesauce. Next was _invoke_state. I spent a little time digesting the Class::MOP documentation (which is quite good), in order to figure out how take a given string for a state/event to be invoked and actually do that. And it was pretty dumb simple: find_method_by_name(). Follow the normal POE::Session behavior of sending things to _default if there is no event found and we are set.

Proof of concept complete. Time to flesh it out a bit. I wanted to be able to change events at runtime. So what does the Kernel do to make that happen? It calls _register_state on the Session. Groovy. I go back to the Class::MOP docs to figure out how to add a method to a class: add_method(). Well that's just too damn easy! I expand my little proof of concept tests to include using $poe_kernel->state('new_state' sub {}); and everything is peachy keen. Except, what about removing? Ah, look. remove_method(). Awesome.

I go off to parade my effort around and that is when Rocco Caputo points out a giant glaring naivety: I am changing the actual class, not the instance that Session represents. Adding a state to one instance effectively adds it to EVERY other instance. Holy leaky Sessions, batman! So back to the drawing board.

What about subclassing? Anonymous subclasses per instance. Sounds evil. Will it work though? I give a whirl. My 'new_state' test is passing. My 'remove_method()' test fails. WTF? I double check the meta object. Yup, has_method() tells me that my anonymous subclass does not have that method. But of course, further up the inheritance tree, my parent does. So find_method_by_name and, hell, invoking the method /works/. At this point, I am bitching. Both ruby and ecmascript support per instance manipulation of methods. Sigh. So I am desperate. I travel to the land of #moose and wail and generally gnash my teeth. That is when Chris Prather had this bang up idea, so evil, so magical, so awesome, that he deserves full credit for it: anonymous class CLONES. Not subclasses, but actual, full out, clones. I consult Class::MOP::Class and Moose::Meta::Class. I see all of the tools available for what I need.

Queue generic john williams evil march music.

So while I was doing this, I stumbled upon a horrible side effect to my evil, evil, subclassing/cloning ways: it breaks POE. So I do some spelunking into the guts of POE to figure out what is going on. Come to find out, POE uses the stringification of your Session object to basically track it internally. And since my cloning creates an anonymous class and I rebless the Session into said class, my stringification changes. Not all of it mind you, because the memory address is still the same. I still have the same object, it is just wearing a bunny suit instead of a santa claus suit.

Ultimately I was breaking POE encapsulation anyhow because I needed to store the newly reblessed Session object into the current slot, but down the road, the mismatch between the stringifications was killing me. And so the evil gets deeper. I overload "". Add an attribute to the Role to store the original stringification. But everything was still broken. After some deduction (print statements instead of the debugger because the debugger was giving me a SEGV, for the lose), I determine that while my sub routine for overloading stringification was indeed composed into the class, the magic of its execution was not happening. Hrm. Querying #moose again returns positive: Shawn Moore notices in the source of overload.pm that it does some symbol table hackery to enable it's magic:

${__PACKAGE__ . "::OVERLOAD"}{dummy}++;
*{__PACKAGE__ . "::()"} = sub {};

And obviously, Role composition doesn't invoke this particular incantation. So I throw it into "after 'BUILD'", and make sure that the symbol table hackery takes place in the anonymous clone classes, too. Now the magic happens in my composed class, which means now my stringification evil works, which means that POE now works, which means it all works! Well, in the simple cases.

There is always more. I wanted to add tests to make sure that composing an anonymous class with the Role wouldn't fail for some weird reason. And so I added that while inside one of the events for the test Session. And, surprise, it fails. But not because it is an anonymous class. I was trying to set an alias on the new anonymous Session from the constructor (the alias attribute has a trigger to call alias_set() on the Kernel) and that alias was getting added to another pre-existing Session, my parent. Silly me. I was still inside my parent's context. POE helpfully switches contexts between Sessions so that when you are constructing a Session, its _start will fire within it's own context, but I was jumping the gun. I didn't have a valid context yet and therefore my alias was getting added to my parent. Add some checks. Make sure to clear/restore my own context after each invocation (to handle parent/child situations). And I am set.

And along the while I also added tracing that looks just like POE::Sessions tracing, but a bit more complicated. Tracing happens on a per method basis and each method actually has around advice applied to it dynamically. I know, not exactly the most performance oriented way to do it. It will likely change to a simple statement just before invocation.

Whew. It works though. You can see the evil arcana incarnate out on github or clone the repo from me here. After I finish up the docs and get it prepared and ready for the CPAN, I'll likely look at doing Real Work with it next. Maybe start an experimental branch of POE::Component::Jabber using it.

On implementing an HTTP abstraction

We all have our little itches that need scratching. This is a narrative on one particular itch that I scratched this past week: POE::Filter::SimpleHTTP.

HTTP. If the criss crossing interconnecting highspeed networks across the world were roads, HTTP would be the unkempt, ball cap wearing truckers transporting our goods.

Now I had a need for such a transport. Specifically I needed to support HTTP in a POE::Filter that could easily be inserted into POE::Filter::Stackable. There were some other solutions out there, that ultimately didn't do what I wanted them to do: POE::Filter::HTTPD only covered things from a server context and only HTTP/0.9, and POE::Component::Client::HTTP was too complex to use when I only needed a filter (Sure I could have ripped the filters out for use, but then the project would have a dependency on the whole distribution when I only wanted the filters). So I guess it was time to write one.

Can't be too hard right? I mean, the world goes 'round with web browsers, and servers, right? So I pull up the RFC and start reading. This is doable I thought to myself. There are some quirks when it comes to Transfer-Encoding, but nothing I couldn't handle.

But I should back up a little bit and give a bit more context on what I am writing, what tools I am using, and what the ultimate goal is.

POE is a framework for creating event-driven cooperative multitasking programs in Perl. It is quite possibly one of the best things invented since hand sanitizer dispensers that are triggered via motion. POE is Perl's answer to other languages' event driven frameworks (Twisted, for example, in the Python world). POE makes it trivial to write large, complex, applications that touch a number of domains. No more dumb select loops. No more event dispatch tables (by hand this is). No more wondering how to merge multiple Net::* modules each with their own event loops. Take look at the evolution of a POE server for what I mean.

POE also provides a number of abstractions that are sweet, sweet sugar. POE::Filter is one of those. POE::Filter defines a simple API for taking in data and churning out transformations, objects, and whatever else you can imagine. And once you have one these awesome little contraptions, you can plug it into a POE::Wheel (which defines the common semantics of setting up IO watchers, etc), and out pops your filtered data.

The idea is to implement your protocol abstractions inside Filters, stack them (via POE::Filter::Stackable [which is a filter itself with the same API]), and plug them into the framework.

Okay, so POE is Mecca. What else am I using? Moose. It is a bit unorthodox at the moment especially in the POE world, but the world view is slowly being adjusted through exploration. Moose is a postmodern object system for Perl 5 that takes the tedium out of writing object-oriented Perl. It borrows all the best features from Perl 6, CLOS (LISP), Smalltalk, Java, BETA, OCaml, Ruby and more, while still keeping true to its Perl 5 roots. It pretty much takes your bland oatmeal and turns into mouth melting awesomeness unbound by the universal awesomeness maximum. There are efforts[ 1 , 2 ] right now to bring Moose to POE in a meaningful way to hide some of the inherent complexity in using POE. And I guess I am now part of the revolution.

And finally to make my trifecta of standing on the shoulders of giants complete, I rely on a few modules from LWP, such as HTTP::Status, HTTP::Request, and HTTP::Response. These are pretty much key, as it means I don't have to implement my own objects to represent HTTP constructs.

And ultimately, this HTTP filter will be only a small part of the greater whole of an XMLRPC client/server implementation. I have a good chunk of the other pieces complete (POE::Filter::XML, POE::Filter::XML::RPC, POE::Component::PubSub), and what was left was the HTTP transport.

So I cracked open the RFC and started digging in. Now, in the POE::Filter context, your only goal is to turn line noise into something meaningful. That means that any socket manipulation should be left for another layer of your application. That said, HTTP/1.1 defines some behavior for dealing with keep-alives, etc which are beyond the scope of the Filter. And so I pretty much ignored it for the Filter implementation.

The RFC defines an army of BNF expressions that represent the textual formation of the protocol itself. BNF is pretty clean, even if you have to jump around a lot in the document to follow a chain of symbols down to the root symbol. I chose to convert, by hand, the BNF to Perl regular expressions. Talk about a great mental exercise. BNF as a grammar has a number of simlarities to Perl regular expressions anyhow, so it wasn't much of a stretch, but there are enough quirks to make it interesting.

As a side note, someone needs to write an Augmented BNF to Perl compatible regular expression translator.

Just to give a taste of some of the regex involved, take a look at the Regex module. Essentially, I built up my tokens and combined them into larger expressions making judicious use of non-capturing groups, character generators, and character exclusions via function.

One special thing to note is that I took a shortcut when it came to URIs and made use of the Regexp::Common::URI module. Only I introduced a bug when I did. I failed to make notice that the Request-Line could take not only an HTTP URI, but also an absolute path. Regexp::Common::URI didn't include this (which I think is a bug), and so I had to implement, specifically, absolute paths from the URI RFC.

But once the regex is complete, the rest of the module is mostly stepping through chunks of data deliniated by network newlines (\x0d\x0a) and keeping track of where we are in the parsing process. That is until we get to the Transfer-Encoding header processing.

The RFC is helpful in that it provides a skeleton of pseudo-code for processing transfer encodings. And that was a great starting point. Something that my Filter does that other don't is the processing of compression in the transfer encoding not just dechunkifying. Unfortunately, this filter is rather simple for this first version, and so the data isn't spooled to disk to avoid OOM exceptions, but will likely happen down the road.

Once we have all of the content decompressed (if it was compressed in the transfer), we store it into the appropriate HTTP::Message subclass and send it on its way.

So where does Moose fit into all of this? The Filter is actually a Moose::Object. And provides a number of attributes for use during runtime for configuration. Also, because it is Moosy, my constructor is generated automagically for me based on what attributes I have defined. No more argument processing. It rocks.

Now here is where it gets cool. Astute readers will have noticed that if I use this Filter unmodified in an XMLRPC stack with the XML parsing filter directly after this one, it won't work. Filter::XML is expecting raw XML, not an HTTP::Message object. So we will have to inject a shim to strip the content out of the message, right? Not with Moose. With Moose, in my stackable implementation, I can subclass SimpleHTTP and apply advice around get_one() and put() to do just that. I don't have to write a container class to call the parent filter or deal with calling SUPER methods that may not resolve correctly if someone else subclasses me or whatever. So many headaches disappear.

But with great magic comes some quirks. Mainly, Stackable has a horrible bug that uses UNIVERSAL::isa as a function to determine that Filters subclass from POE::Filter. Well in the Moose world, I can't get a magical inline constructor if my parent has a constructor defined. And POE::Filter defines a silly constructor. So what do we do?

First, install UNIVERSAL::isa from the CPAN and use it. Second, provide our own isa() implementation that says, yes, we are a proper POE::Filter subclass. Now all of our automagical goodness works as expected.

Throw in some tests that cover our error states, double check the attributes, and make sure our dechunkifying works, write some pod, and push to the CPAN.

And that was it. A new release is on the horizon largely to fix the POD (it's ugly, I know), and to add some more tests (can't have enough of those, right?).

You can follow development of POE::Filter::SimpleHTTP here.

UPDATE:

The second link on the Moose + POE effort was supposed to link to Evan Carroll's MooseyPoopoe on github (http://github.com/EvanCarroll/MooseyPoopoe/tree/master)