Update: Explanation of database partitioning techniques updated and enhanced. See towards the end of the post...Update: Lots of people are coming here from the google-code blog. You might want to check out my posts on Simplicity, Management, Software Development, Career, or the main blog. If you like the content, you can subscribe to the feed. Thanks.
Learning that Google internally uses the
Hibernate Object-Relational Mapping (
ORM) framework brought my opinion of Google
down by quite a few notches.
I strongly believe that
in performance-critical applications, database transactions and SQL queries are best written and managed by the developer rather than by an over-generic ORM framework. If you have a
low-usage application that will be simultaneously used by at the most 300 users, then go ahead, use an ORM layer and knock yourself out. But it is likely that if you have written a good app, it will become more and more popular, and you will start hitting performance problems soon. Any Google consumer application would be at the high end of performance-criticality requirements.
The reasons that have shaped my views are as follows -
-- Typically any application spends 25% of the time executing application code, and the other 75% executing database queries (Rough estimates based on my experience)
-- So if you want a high-performance app, optimize the data access layer
-- An ORM tool is very generic. It is meant to support multiple RDBMS systems and a huge variety of usage scenarios. So these are optimized for flexibility and not performance
-- Optimizations on the data access layer are best achieved by optimizing the SQL.
You know your domain model and data schema best. So you are the best person to write the SQL
-- ORM tools like Hibernate do not allow you to write your own SQL. Instead you give definitions of your database tables and map them to your business objects. Hibernate automatically generates SQL from these relationships
Many people take issue with this and say that to manage transactions and write optimized SQL, you need very good developers, who are expensive and difficult to attract and retain.
My response: Of course, if you need to create outstanding apps, you need good developers. If you can't attract and retain them, that's your management problem. As for the cost aspect, hire a great developer, fire 5 average ones. You will get more work done, and save money.
Anyway, all the above is deviating from the main topic. Which is that
Google has open-sourced Hibernate Shards, which is an extension to Hibernate to enable Horizontal Partitioning of Data.
What is Horizontal Data Partitioning? Well, if your app requires access to large large amounts of data (hundreds of millions of records), and performance is critical, and you have a large number of users, then storing all your data in one database will hit problems. This is because there will be too much read/write to the same database storage. One solution is to
split the data in a big database table into "horizontal partitions" based on some particular criteria, and store the partitions in separate databases. For example, you will group your user profile table based on the user's location - US, Europe, Asia - and store each group of profile data in a separate database. This is called horizontal data partitioning as you are splitting the data at the
"row level" within a database table.Another option is
Vertical Data Partitioning. Here, you partition the data in a database by putting different high-volume database tables into separate databases. For example, if you have lots of User Profiles, Messages, Transactions etc etc in separate tables, then with vertical partitioning, you will store each of these tables in separate databases. This is called vertical data partitioning as you are splitting your data at the
"table level" within a database.
Great Post-It ad. See more here.
