I had a call with Monty the other day and I told him why I think MySQL Server Quality will never be the same again. I’ve been thinking a bit more about it and here is the extended list.
In particular I think MySQL Server will never be able to reach its original quality guidelines (see previous post) and even current release criterias will unlikely be ever reached with any sensible definition of what serious bugs are.
Large Team, Tricky Code During MySQL 3.23 and MySQL 4.0 team the small tight knit team was working on the server and Monty was personally reviewing all the code and knew all the code well to know all the side effects etc. Now MySQL Development is done by different team working on the different components, while MySQL was not designed to be very modular with clear set interfaces and no side effects on the first place to support such development concept easily.
Increasing Complexity Each new MySQL version is getting more and more complicated. MySQL architecture has to be additionally complicated to support modular storage engines with all their interactions and different specifics. Think for example how many DBMS need to do Distributed Transactions (XA) internally just to keep different components in sync. This is multiplied by the fact a lot of MySQL features were implemented in marketing rush being hacks over architecture which was not intended to support it. Remember early MySQL versions knew only about table locks and were not initially designed to support transactions. Most of other databases are designed with these concepts at their core.
Increased User Base I’m not sure if MySQL user base is growing any more but it surely has grown dramatically since MySQL 3.23 or 4.0 times. More users means they naturally find more bugs. This is actually has the good time too. Even though a lot of bugs being found and reported only small fraction of them apply to any given user. The bad thing – in many cases we do not really know which one apply to us 
Low Feature Usage Density MySQL 3.23 was simple so Most of its features were used by majority of users. This was even the early Marketing strategy for MySQL. Marten used to frequently say Oracle is Overkill for most users and most users use only 10% of features or less. You know what ? MySQL is becoming overkill for increasingly large amount of users too. A lot of people in the Web world would be as happy with MySQL 4.1 if it would be actively maintained now. And hey this is exactly one if the reasons behind Drizzle project.
Lower usage density means features are less tested by community during release and they also mean they get less attention from developers, because development team (and especially its efficiency) is not growing on the same pace. This also means fewer customers use the same features, and so the revenue associated with feature is not high enough to keep all the bugs reported for this feature. So bugs sometimes stay open for years.
As an example of low usage density features I would mention MySQL 5.1 features. During my talks I often asked people how many of them use 5.1 vs row level replication or events. I think less than 5% are using both features. MySQL 4.1 was out for years and yet I would say no more than 5% of users use prepared statements.
Bug Reporting and Tracking The MySQL 3.23 received quality was at some extent because there were no bugs.mysql.com – bugs were reported to bugs@lists.mysql.com which meant both it was more complicated for users to report a bug – so fewer people did and it was easily to lose the bugs, especially boring and less critical ones. Monty did a very good job keeping track of these but still not as good as when bug tracking system was implemented and so bug would never disappear unless you fix it or call it a feature
MySQL Enterprise Madness For years MySQL QA policy heavily relied on community and I think it is still the case. This means MySQL latest code was instantly downloaded and tested by thousands of users which would find bugs quickly. With MySQL Enterprise it is customers who got the bleeding edge code and for community the binary which is over half a year old was often offered.
Enterprise customers who got the latest and the most bug free/buggy code (less old bugs more new ones) are fractions of percent of MySQL User base and they are also more conservative – few just would grab latest release as soon as it comes out. Many will only upgrade on the schedule or keep on the same release until they run into some issues. This means bugs in Enterprise version have tendency to stay for long time.
The good thing about this item is – it is easily fixable. Just get somebody in his mind to define release policy, and better have him/her understanding engineering not just marketing.
Innodb/Oracle relationship complexities. If the bug happen to affect Innodb things become even more complicated slow. Even though MySQL and Innodb is tight knit system these are two different code bases controlled by different companies. MySQL can’t just do changes to Innodb codebase and Innodb can’t do same for MySQL. Both require other side approval. This is why I guess all new Innodb Development is going with Innodb Plugin which is separate from Innodb included default MySQL. Why is not this part of stock MySQL 5.1 ? I guess because MySQL 5.1 release was closed for any significant changes for years.
Now how it all affects bugs an not features ? Well if you want to use latest Innodb you will need to use plugin which is not part of standard MySQL distribution so it, just as MySQL Enterprise, gets significantly less testing by community than standard MySQL.
Another problem for Innodb Plugin quality is lack of the full disclosure about bugs. Check this change log for example. You would see some bugs fixed have bug numbers and large number does not. This means there are some bugs which are discovered by Innodb/Oracle team but they are not disclosed via searchable public bugs database.
I should note besides all these process issues Innodb remains remarkably high quality component. The initial version of Innodb Plugin was very good in terms of quality for version having first public release as we tested it back in April.
The Good News in the End
The good news are you really do not need general “quality” or “being free from bugs” to run MySQL in production. MySQL as it is now works for a lot of applications, in particular because most of them use only small portion of MySQL features.
The post 7 Reasons why MySQL Quality will never be the same appeared first on MySQL Performance Blog.