The WordPress Theme Review Experiment – Take II

About a year back I wrote an article on the Theme Review process that WordPress had instituted. In only its second month at that time I was among several users to have expressed frustration with the process. Several months have passed since the experiment has moved from the lab to the mainstream, and I thought of taking some time to revisit the process to see how it has helped or hindered the reviews.

Significant Improvement in Quality

The biggest outcome of this effort has been a significant improvement in the quality of repository-hosted themes on http://wordpress.org. In the initial stages the process was aimed mainly at catching themes with spam links and obfuscated tracking code. Gradually, as the reviewers found their feet a lot more standards and policies came into force. The review team, or WPTRT (WordPress Theme Review Team) has been steadily enforcing a lot of these:

1. Repository-hosted themes are required to have zero debug errors

   What does this mean? If you look through your server logs you shouldn’t see any messages that say “NOTICE”. WP provides a much easier way to check this. You can go into the wp-config.php file of your installation, then put in this line there:

   define('WP_DEBUG', true);

   If your theme doesn’t spew a lot of messages in the admin panel upon activation, it is probably clean. Of course, you need to also check out each type of view in the front-end, like the search page, a single post, custom widgets etc. Not having error messages is not only good practice, but it also improves performance, as your web server is saved the CPU cycles of writing the error to the log.

   Here is the kicker: setting this flag will almost immediately spit out errors for plugins, because most plugin authors have probably never coded with this flag on. That is why on my DEV environment I rarely have plugins activated.

2. The Theme Review criteria are kept up-to-date
   Every theme has to go through an elaborate review process in terms of standards and compliance. The criteria are divided into categories such as what is required vs. what is recommended. Some people do complain that the criteria change too frequently. However I have managed to stay abreast of the changes most of the time (plus I run a periodic check of my theme – see below), so this has not been an issue for me since quite a long time.
   The review criteria cover various aspects, key among them being the use of standard WP calls. Basically WP’s philosophy is “Decisions not Options” – themes are intended to work out of the box. In other words, a theme like Suffusion is going to be looked down upon by WP aficionados (because it really gives you too many options). But I digress. The expectation is to have every theme be extensible by child themes. The use of standard functions helps this process significantly. In case you were wondering, Suffusion makes use of almost every one of those (apart from WP’s standard Custom Header, Custom Background and Editor Stylesheet functionality).
   Every developer is advised to go through the review criteria to ensure that his theme passes the checks before handing it off to the review team.
3. Themes are tested against a complete test suite
   Something that was rarely updated prior to the WPTRT’s setup has been getting quite frequent facelifts. There is a full-fledged test suite frequently updated to account for WP core enhancements. Developers are encouraged to install the latest versions of the test suite when available and ensure before submission that all the tests pass successfully. The reviewers have to ensure that all the tests are passed while doing a full review of the theme.
4. Automated tools are used to check for bad stuff …
   For all the hours of effort that must have gone into the Theme Check plugin, the benefits are invaluable. Any WP developer can install this plugin, then run it against his installed themes. It provides details at several levels:
   1. Deprecated function calls with suggestions for new functions
   2. Missing usage of required functions
   3. Code obfuscation and spam links
   4. Potential security loopholes
   5. Basic checks like licensing information

   This is probably the most useful plugin a WP theme developer can have. The tool was made a formal requirement for theme submissions about a couple of months back, so any theme submitted would automatically be checked using the tool and the developer would have to ensure all checks are passed.

5. … And the tool developers and maintainers are open to feedback
   There is a constant refinement of the process. To cite an example, I was using a function called get_users_of_blog, that got deprecated with WP 3.1 in favour of get_users. Now, if I was to switch to get_users_of_blog in my code, people still using WP 3.0 would run into issues – after all it was quite natural to have users stay on the previous version of WP to avoid plugins breaking. But if I didn’t do the replacement the theme wouldn’t pass the check. So I highlighted this issue to the review team and a resolution was reached quite quickly: deprecated functions that are one version old will be given a pass. The plugin was promptly modified and further discussions were averted.

Turnaround Time Close to Pre-WPTRT days

The WPTRT came up with a process of classifying submitted themes into 3 categories:

1. Priority 1: Themes that have already been approved. Themes in this queue are addressed first. Such themes typically go through a “diff” process, where the reviewer checks the changes against the previous version and if the changes look okay, the theme is approved. Of course, there are exceptions when a full review might be required (I have had it happen when I redid Suffusion’s innards in 3.7.7). There is some ambiguity as to what happens if the reviewer finds bugs or errors. The theme gets pushed to Priority 2 (see next) upon resubmission, but if the resubmission happens within a reasonable time-frame the original reviewer can review it as if it were a Priority 1 theme.
2. Priority 2: Themes that have been reviewed at least once and the latest version was not approved. The original reviewer marks the old submission with information regarding how complete the review was. If the original reviewer had said that the old review was complete, the new reviewer simply needs to verify that all the points in the old review have been addressed. If the original reviewer had said the old review was not complete the new reviewer needs to check for the old items as well as potential issues with the new submission.
3. Priority 3: New themes that have never been looked at by any reviewer.

With an increased number of reviewers and a much more streamlined process, the review time for previously reviewed themes has definitely come down, though I don’t have exact statistics.

Trainee Reviewers

Any time that a new reviewer wants to start, he is put through an incubation process as a trainee, where he looks at an assigned theme (typically a Priority 3 theme), makes comments and submits it to someone who is a full-fledged reviewer. The latter reviews the feedback and makes comments on the thoroughness of the trainee’s review. After a few such cycles when there is enough confidence that the trainee can go out on his own to review themes, he is upgraded to a full reviewer. This ensures that every reviewer is fully aware of all the areas a theme can potentially trip, and every review is of good quality.

What Could Improve

Obviously there is always scope for improvement. Being an open forum the WPTRT mailing list often sees animated discussions on what should an shouldn’t be there in themes:

1. Theme vs Plugin Discussions
   Very often a theme developer includes some functionality that has little to do with the look and feel of the theme. Classic scenarios in this case include combining scripts on the fly to optimize site speed, or include SEO options in the theme. The developer gets told each such time, “That is plugin territory”. While that may be true on the face of it, the heart of the matter is more complicated. A lot of developers release themes in the repository to compete against the better marketed “premium” themes. There are many reasons for this, primary among them being strong advertisement across the web by folks including core WP members about using repository themes. If a new user comes to the repository based on advice off the web, then starts comparing the popular themes in the repository with top “premium” themes, it might soon start falling apart. For, all the standards apart, an end user will see much more value offered by a premium theme simply because the so-called premium themes have no boundaries to work within. There are a few ways to work around this, none of which is easy though. One would be the following:
   1. The theme page at WP (such as http://wordpress.org/extend/themes/suffusion) could be made to have a list of dependent plugins that the theme works well with or recommends.
   2. Upon downloading the theme the corresponding plugins get downloaded as well. And upon uninstalling the theme there can be an option to deactivate the extra plugins. Of course, this will require probably significant changes to the theme administration in the core.
   3. The biggest challenge with this setup, though, is that plugins have close to zero code quality control. All popular plugins not distributed by Automattic have debug traces that make you shudder. And plugins are often written in a manner that is not optimal from a security point of view. Given that the total number of plugins is orders of magnitude higher than the number of approved themes, getting a manual process around plugin approvals is foolhardy.
2. Specialized Theme Support
   This is a gray area whose resolution will be of significant benefit to developers and users alike. There was a repository theme called Quality Control, which helped users create a bug-tracking system in the standard WP framework using Custom Post Types. The theme is no longer available on the repository because the developer got paid to develop this for a company. You can still get older versions of the theme (which are licensed under GPL). The theme displayed a very innovative use of WP. Unfortunately though, since it requires the use of Custom Post Types in the theme itself, users are compelled to stick to the theme if they want to have their content available on their installation. Basically the user gets into a “lock-in” situation with the theme.
   There was a suggestion to have the authors of such themes provide a way to opt out of the theme. In case of the Quality Control theme it would involve the visual provision of the code to generate the structures of the post types and taxonomies. Users could then paste the code in a new theme and make it run there.
3. Vocal Opposition to Options
   This is a weird one. It is understandable that themes should be extensible by means of appropriate hooks so that adventurous folks can write the code themselves in child themes. But of WP’s several million users, barely 1% can figure out what the anatomy of a theme is, the multitudinous tutorials and oodles of documentation notwithstanding. It is surprising that having code-intensive frameworks is encouraged, but a UI-intensive framework that achieves similar results is frowned upon. Are all users expected to get into PHP code and make a child theme if they wish to display something like excerpts instead of full posts? Admittedly the percentage of people opposing themes with elaborate options panels is small, but if you were to survey people across the web on what they would like better, I am sure you will find that themes with more options have more takers.

How I Have Improved

I have managed to keep abreast of WP changes by various means. Occasionally I go through the last few reviewed themes and see what kind of comments came up from reviewers. I then do an introspection to see if I am breaking any guidelines myself, then I embark on making the appropriate changes.

The ultra-buggy 3.7.5/3.7.7 release of the theme actually involved one such fundamental paradigm shift – I switched to using the Settings API. Nobody had mandated the shift: it just made sense based on comments provided to others. Getting the back-end coded to work with the Settings API was a huge challenge and I spent many hours labouring over getting things right. This was specifically because I had so many options that I needed to group them in three levels (most tutorials on options deal with just one level, and an exceptionally detailed one by Chip Bennett deal with 2 – you can see my comments there).  But at the end of the day it was quite satisfying to get the API working and in place.

Another key functionality I built in was an image resizer, again in 3.7.5/3.7.7. I used to have TimThumb as the resizing script, though I did have support for native WP thumbnails as well. But with the 3.7.5 release I completely eliminated TimThumb, thanks to repeated discouragement for the script in the theme reviews. My resultant code is quite powerful and versatile, but is limited by certain ways of WP handling image resizing natively.

As it turns out, there are a few things I still need to address in Suffusion, namely the use of WP’s native Header and Background functionality. I will be pondering over those the next few days, because the last thing I would want is to break functionality that is working fine for the thousands of users of the theme.

But believe me – it makes a developer very happy to have zero errors on debugging, and have Theme Check say “Pass”!

When in Doubt, Ask!

If you are implementing a big ticket feature that can trespass into gray areas of theme development and involves writing to files or reading from files, do yourself a favour. Ask. The WPTRT mailing list has folks like Chip Bennett and Edward Caissie (Cais) who have been a part of the review team since its inception, Simon Prosser who has written the Theme Check plugin (currently maintained by Otto and Simon), Otto and Justin Tadlock who are WP veterans, and Andrew Nacin who is a key member of the core WP development team. There are other volunteers who have been a part of the process for a very long time.

You might not be in 100% agreement with what they say (a lot of it might appear to be stifling if your goals are ambitious), but asking before implementation will give you fair warning and a potential sense of direction with respect to how you should implement certain features. It will also help avoid some long-drawn and nasty battles.

To Conclude

The Review Experiment is no longer an experiment. It is a formal process that is here to stay. It has helped improve theme quality across the board. Kudos to the review team for making the process successful!