A disclaimer
I would have posted this on WP’s official site if it allowed me to post, but since it doesn’t, I am publishing it on my own blog. This is meant as an appraisal of the new theme approval process on WordPress.
On 9th June 2010 WordPress made an announcement that it was expanding the theme review process. There was a call for volunteers to support theme reviews. This was indeed good news, particularly if you were a theme developer. For a long time Joseph Scott had been the sole approver of themes on WP, and though his work was impeccable and his reviews very good and accurate, if he was traveling or unwell or on holiday theme approvals tended to get delayed. This was by no means a deal-breaker, but having more people looking at themes definitely meant that things would get better.
Soon after the announcement a few things happened that were moves in the right direction:
- Quite a few folks signed up in response to Joseph’s post, so there are more volunteers now.
- A dedicated Trac project was setup for approvals.
- Every time a theme got submitted the corresponding Trac ticket got emailed to the theme author so that he/she could follow what was going on with his theme.
- There was much better focus on the quality of code in a theme. Actually the focus was more on writing code that didn’t generate “silent errors” or PHP NOTICE messages, but nonetheless adhering to this is something that keeps down the size of logs on your server.
But that is where the goodness stopped. One of the key things which I had hoped would be addressed, speed of approvals took a nosedive. To cite an example earlier when I submitted a theme version it would take around 1-3 days to get approved. It took 1 day if I submitted it on a weekday and typically 3 days if I submitted it on a Friday. In some cases if it was vacation time then the versions took about a week to 10 days to get approved, but this was more of an exception to the rule. Normally on a day you would see 2-3 versions of different themes getting approved, and on occasion the number of approvals even ran into double digits.
However, with the new experiment somewhere something went wrong. Now you have around 1 theme being approved in 2-3 days, which doesn’t seem right. To drive my point home, version 3.5.7 of Suffusion got approved on 10th July. Today is 28th July and only nine more versions of different themes have been approved (including a version of the WP default theme Twenty Ten). This is a far cry from how things happened earlier. I normally used to wait for about a week before pinging the folks to check the status of a version, so for 3.5.4 I sent out a message asking what was up, and this is what I received:
Thanks for the note. Unfortunately there is currently a backlog of themes in the review queue. We are working on it. Themes are being reviewed in the order in which they were received. Thank you for your continued patience.
I didn’t know at that time how much the backlog was or how much longer the next version of Suffusion would take for approval. I found out later how you can estimate the potential wait time. The backlog, as I know now, is well over 100 theme versions at present. So somehow adding more people slowed down the process.
The delay also brings in a bunch of associated problems. Currently if you have a theme version in the queue and you submit a new version, the older version gets deleted and the newer version goes to the back of the queue. So as a theme developer:
- If I have submitted a bug-fix for the theme 3 weeks back and that hasn’t been approved, how do I ensure that my users get the bug-fix? Distribute it from my site (unapproved)? Doesn’t sound safe.
- If I have submitted a version 3 weeks back with some bug-fixes, then I discover some more bugs in the submitted version what do I do with them? If I resubmit a new version I go to the bottom. So I add another month or so to something that has already been in for 3 weeks? Ouch. Incidentally I have had this happen to me quite often, most recently with version 3.5.8 and 3.5.9.
- If I choose not to submit the bug-fix for the older version but let the bugs be approved (these are niche bugs that cannot be caught in the approval process, mind you) and then I submit the new version, I am risking users staying with a buggy version for an indeterminate number of extra weeks.
So somehow this review process discourages active theme development, which I feel is counter-productive to the whole experiment.
How can the experiment be improved? I have a few suggestions:
- Improve targets. Set a target saying that each approver should aim to disposition a new theme for approval every weekday (or some such figure). If you have 10 approvers, this will take care of tackling 10 themes everyday and bring down the backlog much faster.
I can almost hear people say, “What does he know?”, or “He is speaking like a consultant”, or “He is oversimplifying”. Well, I have been doing free support for Suffusion since late last September and that is how I keep my backlog down. Every morning I go through all outstanding support requests, take a crack at answering them if they haven’t been answered by someone else, then carry on with my regular job for the rest of the day. If a request comes in during the day, if I am not doing much then I respond to it (or if I think others can respond, I skip it). This has been a largely successful way of staying on top of things. I have responded to more than 3000 posts on my support forum at around 10 posts per day. - Don’t de-prioritize existing themes in the queue upon newer submissions. I love putting out new functionality in the theme, and anyone with any experience whatsoever with Suffusion will agree with me in the fact what I provide is not just cosmetic, but hard features. Sometimes a new version is required to patch a bug. Getting pushed to the back of the queue is very off-putting, because it simply adds so much more in terms of wait time for approvals.
This simply encourages parallel avenues of theme distribution because developers know that their work will not be approved soon on WP’s official site, so they distribute it from their site. And that is where they can start putting in stuff like tracking code etc into a theme. Mind you, quicker approvals will make this a non-issue, but the current wait of a few weeks is a bit too much.
As a concluding remark, the topic of this post and the issues in the post are not mission-critical and I am sure people have more interesting things to do. However I am sure that the free theme developers would be quite happy with these modifications, particularly since some of them invest a lot of energy in building their themes.
Hello
I understand and I agree.
It is not actually discourage good intentions and developers.
I want to say that is a theme Suffusion very successful.
Good luck for the future.
Regards.
Thanks.
Hi Sayontan, out team is new to your theme but we are already using it for a couple of clients. You will be pleased to know the site developers here are all business consultants with no prgramming skills!
We have a question for you. When the Suffusion theme is used in conjunction with Buddypress the sidebars get pushed underneath the Buddypress content but only on Buddypress Pages. However the sidebar reverts to its correct position when navigating back to non-Buddypress pages. Do you have a nice and easy solution to that problem please? Thanks and wow what a fantastic job Suffusion is. Think you will soon pass the other contenders with your ratings.
Hi again Sayontan, your Buddypress Pack solved the problem. The generic BP template pack we were using did not. So problem resolved and again many thanks for whatwe are hoping could become the standard out there!
ps the only think we could add really is the need for a little more speed in the admin area but it sure is not a show stopper.
Rgds Chris
Chris,
Thanks for the feedback. FYI, there is a full-blown support forum (see the top of the page) that you can use for queries – I tend to ignore queries posted through comments or email.
Regarding the speed of the back-end I am aware of it. The issue there is that the humongous number of options are all loaded into JavaScript variables, slowing things down a bit. I have already submitted version 3.5.9 of the theme with significant improvements in the admin panel speed, so hopefully the theme reviewers will approve it soon; it has already spent quite some time on the review queue.
Sayontan.
You are quit right. Slowing down the process in this way is killing for all those enthousiastic people using WordPress and off course their developers. So now what? Do we spam Joseph Scott with our “demands” for speeding up the process ;-). Should we use updates that you put on the web? To be honest, are your updates really getting better when reviewed by others? I feel safe enough by using your not reviewed updates of the theme. Never saw a person on the web with such development speed and quality like you, for free!!!! Your work is very very very much appreciated 😀
Greetings…
Wim
Thanks, Wim! I have to say that the quality of my updates has improved in one aspect, though you wouldn’t notice it. Earlier if you checked your Apache server logs you would see them flooded with PHP NOTICE messages. That number has come down significantly.
[…] This post was mentioned on Twitter by Alexandre Enkerli, corsetkitten. corsetkitten said: https://aquoid.com/news/2010/07/improving-the-wordpress-theme-review-experiment/ #wordpress #wp […]
Hi Sayontan,
Great feedback on the Theme Review process (even if I’m a few months late to this post – I just discovered it today). I just wanted to post a bit of a response, and a follow-up.
First, in defense of the Theme Review team: it took us a couple months really to get up and going. First, we had no good, objective list of review criteria by which to ensure that all of us would be conducting fair, objective, and consistent reviews. It took us at least a month to develop such guidelines.
After that, the Theme Unit Test data needed to be updated (the previous XML file was from 2008).
Even then, each of us had to conduct several reviews – making several missteps in the process – before we really felt like we had our feet underneath us. Joseph Scott had done yeoman’s work in handling all this by himself before forming the team! We found that we had a considerable learning curve before we were really performing good, consistent reviews.
Your point about cycle time for Theme review and approval is an ongoing problem. The sad reality is, the Theme Review team only has 3-5 people who come anywhere near approaching an average of 1 Theme review per day, while Themes continue to be submitted at a rate of around 10 per day. It is a constant struggle, and we’re continuing to find ways to expedite the process. To that end:
1) We have implemented a couple of the things you suggest here:
First: tickets for already reviewed Themes are prioritized ahead of tickets for newly submitted Themes, and rather than a full-blown review, are given a review based on the diff between the approved version and the new revision. This way, bugfixes and other improvements to already approved Themes should happen much more quickly.
Second, for all tickets, if a newer version is submitted before the existing ticket is reviewed, the new ticket takes the original ticket’s place in the review queue. This way, bugfixes and other improvements do not cost the Theme its place in the queue.
2) The Theme Review continues to add tools and helps for Theme developers:
First is Simon Prosser’s Theme-Check tool, that will check Theme .zip files for issues that would cause the Theme to fail the review.
Second, the Codex pages are continually improved, and the Guidelines continually revised, based on Theme-developer feedback.
3) We’ve attempted to improve the submission and review process, by making better use of the tools available to us. We continue to improve our use of Trac, adding version-diffs for previously approved Themes, adding the Theme developer to the ticket by default, to ensure email notifications work, and have created the review prioritization reports. We are right now in the process of improving the Theme uploader script, to ensure that it checks for more criteria, and outputs all results in one pass, rather than requiring multiple passes.
If there is anything else that anyone can think of to help improve the Theme Review process or guidelines, we’re always open to feedback. We’re trying to do our best, but please let us know where we can do better!
Thanks,
Chip
Chip,
Thanks for the extremely detailed response, and I am glad that this caught the attention of someone on the theme reviewers’ team.
I did notice that the last couple of releases that I made got approved extremely quickly, within a day or two and I made a point of mentioning this on some of my release announcements:
So obviously themes previously approved are getting the “frequent flier” privileges, which saves a lot of time.
I guess overall once the speed issues are ironed out this will be a really great process. I know for sure that Suffusion is much more adherent to good design guidelines as per WP recommendations and if all themes follow the same rigorous approval method, the overall quality of themes on WP will go up significantly.
BTW, Simon’s tool sounds pretty interesting. Is that a variant of the current upload checker (or is this already in action)?
Thanks for taking the time to respond here.
Sayontan.
Thanks for the mention in the 3.6.5 release notes! I’m glad an important bugfix was able to get out to your users as quickly as possible.
Long-term, what the Theme Review team really needs is more volunteers. In addition to the sheer volume of Themes submitted, we also have to deal with the complexity of some Themes. Speaking for myself personally, I’m not particularly expert in JavaScript/jQuery/etc – so it takes me longer to perform a proper review of a heavily scripted Theme. Also, we get a fair number of BuddyPress themes – and we only have one person on the Theme Review team who has much expertise with BuddyPress.
Hopefully, the addition of more automated checks during the upload process, and general education about the guidelines, as well as the overall ironing-out of the review process, will make things easier for everyone, moving forward.
As for Simon’s Theme-Check tool, it’s available for anyone to use. We’re in the process of incorporating most/all of its checks into the Theme Repository uploader tool. Hopefully that should all be ironed out within a week or so!
Well here we are at the start of 2011 and if you look at the backlog, there are themes waiting to be reviewed that were submitted 5 weeks ago. So instead of speeding up, it looks like things are slowing down.
Anyone know what the criteria are to judge the ‘WordPress theme review Experiment’ are?
Indeed, there is a backlog. But you might consider that we have just gone through the holiday season. All of the reviewers, who contribute entirely on a volunteer basis, have families and full time jobs that don’t involve WordPress or Themes. A backlog – especially around Thanksgiving, Christmas, and New Year’s – is to be expected.
You’re welcome to join us, and help take care of that backlog!
How about putting up a donation box in wordpress home page to support the theme/plugin repository. So, the theme reviewers won’t work as a volunteer.
By the way, I found this website. Is it a official blog of wordpress?
http://wordpressfoundation.org/
Just my two cents
Towfiq I.
[…] a year back I wrote an article on the Theme Review process that WordPress had instituted. In only its second month at that time I […]