Sunday, 2 September 2012

Complex web pages have huge differences in rendering time in different browsers

I recently had the task of optimizing HTML for a web application that was getting somewhat slow. We were seeing big differences between browsers. Chrome seemed to be exceptionally fast. So I went ahead trying to reproduce this performance issue in order to decide where to go ahead with optimization. 

I knew that the web app had lots of nested divs with a lot of inline styling. I was curious to find out what performance penalties this had so I made this client side performance test.

There's a lot of websites that talk about website optimization and they all say you should minimize the payload that has to go over the wire. I could not find much on the topic of speed when it comes to rendering large HTML documents.

Summary of results
My tests show that large DOMs with many nested elements (in this case 7381 divs) can have a huge impact on client side performance. Compared to Chrome, IE9 spent 9-25 times longer rendering the exact same page.

The tests
I programmatically created 3 html documents; A, B and C. They all have nested divs and some styling. They look something like the grid below. It has 1 big square and inside this square there are 9 squares and inside this there are 9 squares and so on. It goes 4 levels deep so there's a total of 7381 divs. Fairly simple HTML document but the DOM is relatively big.

Measurements where done using the HTML documents that are linked to blow. The server was located in the same country and the Internet connection where this was carried out was very good. The browser cache was unprimed. All the divs have fixed size so drawing the layout is pretty easy for the browser. There is no need to recalculate and re-position elements as the page is gradually received.

Test A 
This document has lots of duplicate inline styles just to create lots of bytesize. Each div looks something like this:
<div id="d_34" style="width:34px; height:34px; float:left; dispaly:block; margin:0; padding:0; border:1px solid #FFFFFF; background-color:#bbb">...
Test A document size is 1.1Mb

Typical time until received all bytes390ms330ms339ms
Typical time until DomContentLoaded event fired 2,22s411ms1,33s
Typical time until OnLoad event fired2,43s408ms1,34s
Time from all all bytes received to onLoad and DomContentLoaded event fired2040ms81ms1001ms

Test B 
This is just the same as A only with all the inline styles moved out to an external css file. Now each div looks like this:
<id="d_34" class="lvl2">...
Test B document size is 237Kb with an additional 400b CSS file.

Typical time until received all bytes187ms178ms195ms
Typical time until DomContentLoaded event fired 1,02s227ms366ms
Typical time until OnLoad event fired1,18s210ms457ms
Time from all all bytes received to onLoad and DomContentLoaded event fired993ms49ms262ms

Test C
This is without any styling at all. Each div looks like this:
Test C document size is 179Kb.

Typical time until received all bytes172ms130ms160ms
Typical time until DomContentLoaded event fired 468ms175ms170ms
Typical time until OnLoad event fired590ms140ms253ms
Time from all all bytes received to onLoad and DomContentLoaded event fired 418ms45ms93ms

What does all this mean anyway
From the numbers we see that when it comes to comparing browsers, Chrome outperforms the others. IE seems extremely slow when it comes to rendering this particular markup. Although IE9 is credited for being much faster than earlier versions, it is clearly far behind Chrome for this specific test.

Chrome is particularly fast when it comes to rendering. It only uses about 50ms to render the page which consists of more than 7000 elements. Chrome is faster for all cases (with inline styles, with external css file or even without any styling at all).

The test also confirms that avoiding inline styles on elements will increase performance considerably. The reason seem to be not just that there is fewer bytes to transfer but that it is easier for the browser to parse the document.

Please note that our test is not representative for common web pages or for other web pages for that matter. Browsers use different algorithms and techniques for different operations such as receiving, parsing, calculating layout, drawing etc.

Saturday, 18 August 2012

50 reasons why I don't like SharePoint development

I was introduced to SharePoint in 2001 when it was called SharePoint Portal Server 2001, codename Tahoe. From time to time I have been put on SharePoint projects but I never came to like it.
I think I have worked on more than 10 SharePoint projects, some of them fairly big. The last time I used SharePoint was back in 2011 with SharePoint 2010.

This blog post is not intended as an evaluation of SharePoint with a socio-technical economic corporate or whatever perspective. SharePoint might, or might not, be a good choice for your development project. This post is Not about SharePoint as a viable product. This post is solely about what I think about SharePoint as a development platform.

Please note that I am talking from my experience as a developer who happen to like simplicity, structure and who cares about usability. What you will find here is simply my personal view of SharePoint development. This is based on doing deep custom SharePoint development, not just configuring out-of-the-box SharePoint sites. Also note that this blog post was written before the release of SharePoint 2013.

Not just me being an old crank
Even prominent people within the .Net community such as Carl Franklin and Richard Campbell openly say what they think of SharePoint. In .Net rocks podcast episode 734 Richard Campbell makes the following joke: "The thing about SharePoint is you start on having a problem you think you can solve with SharePoint and then you have two problems."

Richard Campbell even goes as far as to say SharePoint makes you cry.

You'll even find developers of SharePoint at Microsoft out right saying: "It is not, and never will be, a viable developer platform."

The 50 reasons why I don't like SharePoint development and why I'll hopefully never do any more of it:
  1. SharePoint kills creativity with its configurations. Flicking on xml/xsl files is not very inspiring.
  2. The backend user interface is mostly not very intuitive. I as a developer don't like to develop systems that suck.
  3. SharePoint as a framework is too big and complex and you never get to feel that you master it all.
  4. The product is pricy and too big so I can't use it for my private projects. That means I will not be able to reuse the skills on my hobby projects.
  5. SharePoint experience has limited transfer value for other type of development projects. 
  6. There is usually IT politics involved since it's a big decision to start building the competence and infrastructure. With politics often follow dirty games, power struggles etc.
  7. SharePoint requires lots of training and long-lasting changes in the behaviour of users. This means that you don't get to see the results of your work and how users actually end up using the system. By the time the use of the solution has stabilized you are long gone on other clients, other projects or have left the firm.
  8. Once an organization and the development team has acquired SharePoint licences and competency they tend to reuse SharePoint on other IT projects where SharePoint is not the best fit. SharePoint often becomes overkill and results in very large development costs. You as the developer may end up having developed a system that is doomed to fail. 
  9. Organizations who have acquired SharePoint for their internal systems often want to reuse the platform for their public facing website to consolidate their infrastructure/licenses. It brings with it a large overhead and the public facing website becomes too complex to manage , maintain and evolve. You as a developer might get stuck doing boring support or very difficult maintenence.
  10. Agile methods can become problematic. SharePoint content types cannot easily be changed once it's in production or even in testing environments. This means projects have to be fairly waterfallish.
  11. Difficult to write unit tests. Poor separation of concerns.
  12. SharePoint version 2010 does not support .Net framework 4.0.
  13. The platform is resource demanding and becomes slow on your dev machine, especially if you have a VM setup.
  14. Debugging is slow. Timeout occurs. When attaching the debugging you'll often hook on the wrong w3wp.exe.
  15. Poor HTML rendered. Takes LOTS of time to customize rendering.
  16. No Asp.Net MVC support (unless you are only using the SharePoint API) .
  17. Heavy frontend html/js/css that runs slowly in the browser.
  18. Lots of XSL files. I never liked XSLT.
  19. Making control adapters is a headache, must deploy but then reactivate web app feature to deploy app_browsers file. compat.browser with control adapters debugging often don't work.
  20. The ghosted/unghosted feature of content types is a big headache.
  21. Some super users will have the rights to change contenttypes, delete site columns, delete or move lists. This is dangerous if you have custom backend code that uses these lists.
  22. Very difficult to investigate content in the database when debugging.
  23. Often you do not need workflow, content approval and document versioning etc. All these things make the solution more complex and you will have to remove or deactivate these things if you don't need them. Scripting removal/deactivation takes time.
  24. Lots of things happening under the hood that is not documented very well. New developers, not familiar with SharePoint will often make bad choices that will be very hard to fix later.
  25. SharePoint requires a lot of competency so you often have persons specializing i various SharePoint fields and often you'll have parts of the IT department that do SharePoint maintenance/operations tasks. This creates dependencies between people and you often get stuck waiting for other persons or departments to deploy or do other development tasks. If the required person or department is unavailable you are of course stuck and you look bad.
  26. Memory leaks are easy to add (although there are tools to find them), App pools regularly crash.
  27. Packaging and deployment is of course different from anything else. Additional competencies needed.
  28. The sheer magnitude of the product is difficult for developers. You'll need different third party tools. All the knowledge required is overwhelming.
  29. When the site goes down or suddenly become slow after an upgrade it is often very difficult to pin-point the error.
  30. The log file (ULS) is always filled with a ton of entries and you have no idea what half of them are. The ULS log is typically 2000 lines long and the error messages are very bad.
  31. The API returns data that are concatenated and needs to be parsed.
  32. Has to run on Windows server. It can run on Windows 7 but not really straight forward to configure your local environment if your team has a good continuous integration setup.
  33. Unpredictable system performance. When you have performance problems and investigate you will often not find the problem because there are just too many moving parts. Suddenly the performance is better. You look at automatic database tuning, at changes in content and changes settings/configurations. You look for memory leaks, do performance testing in the testing environment, restore different versions of the production database, you look at visitor statistics, web server logs, sql logs, application event logs, ULS logs, look at timer jobs, various manual jobs that can have been started, you look at users behaviour during the period, you check automated response testing like Gomez, what's hitting your site, crawlers, what content is being read, when is backup and indexing on the database running. Has the MOSS search crawler been running etc but the problem is often a mix of interrelated issues so it's hard to find by investigating sub systems one after the other.
  34. SharePoint has too many configuration options. An example is regional settings on each site. You might get problems showing dates consistently and not understand why.
  35. Lots of access control and post release configuration required. You'll have to determine what content types, site templates, page layouts, webparts should be allowed to use by who and where.
  36. If you on a page want to hide fields that are empty or add some conditional formatting you'll have to create custom webparts or usercontrols. Webparts that should be hidden in special scenarios requires additional development.
  37. Some out-of-the-box webparts (such as search) creates poor html which makes them harder to style and look a like on different browsers.
  38. Since everything is customizable in SharePoint you end up with less user friendly dialogs. For example if you use a very simple page publishing workflow there are still going to be superfluous user interface elements and interactions that makes it messy for the user.
  39. Ambiguous error messages.
  40. In a script-it-all dev environment you often configure some part of the SharePoint solution and to test it you do a deploy of the entire site by building it from scratch or running some upgrade scripts. You find that the configuration you just wrote did not work for some reason. The scripts takes 20 minutes to run so you spend the rest of the day tweeking the configuration files to get it to work by running the deployment scripts over and over.
  41. Setting up a sitedefinition always seems to be a problem. Settings up a new sitedefinition and configuring what features, page layouts, content types, master pages, welcome page should be used for a site seems like a straight forward job. But this for some reason always fails and you have to modify the configuration and redeploy and redeploy and redeploy until you get it right.
  42. A single typo in a setup config file can destroy your site and will be hard to locate.
  43. You are using a very large platform and you can't just copy one installation to a new site and start from there.
  44. In most cases you will start by reconfiguring the site from scratch. So most of what you do is to write code or configurations to build and deploy your site instead of spending time developing cool features using cool code.
  45. It is difficult to script everything. There are simply to many customizations and configurations that can be done. There is always some manual change that someone forgot to script.
  46. Too many layers of development. In other projects you might just have to copy the entire site, make some adjustments in a few config files, then set up the database and you are good to go. But with SharePoint you will have to write code that builds the things that builds your solution (unless you are doing simple projects using the SharePoint designer).
  47. The whole development process tends to get messy because things are just hard and complicated with SharePoint.
  48. Creating Internet facing websites are difficult because the platform has a lot of overhead. Lots of effort to go around things, removing things and making the site more accessible and faster.
  49. Because SharePoint development is so complex things very often takes more time than estimated so you miss deadlines or have to work overtime.
  50. If you have a SharePoint project with high expectations on visual design, usability or accessibility then you will require designers with good knowledge of SharePoint. This can be hard to come by and/or will require excellent collaboration between developers and designers. Often challenging.
  51. Microsoft usually makes big changes to SharePoint with each new version. If you have not followed best practices you are in for lots of work if you are upgrading in the future. What the best practices are takes time to learn.
  52. The preferred way to store data in SharePoint is with SharePoint lists. If you do not use lists and instead go with separate databases you can end up with a fragmented solution with many databases (SharePoint warranty breach if you meddle with the internal SharePoint databases). If you use SharePoint lists you are likely to run in to issues with the abstraction layers and problems trying to put relational data in lists.
Further reading

Thursday, 8 March 2012

Native App or Web App for Mobile?

Ok so here's yet another web page on the Native App vs. HTML debate.
Too often I see people making snap decisions without a thorough analysis. So here's another of my blog posts where I attempt to cover all the various factors that should be considered when choosing between a native App or a web based mobile site.
This was written for organisations that don't want to spend the money or time to develop both mobile web apps and native apps. You can of course make both native app and a html web app or mix the two using hybrid approaches such as Phone Gap etc. You can also make dedicated web apps for dedicated platforms.

In the listing below I assume you or your organisations have some experience with web development and you are just considering a native app vs. a general web app. You already have a website designed for desktop.

When to go for a Native App
  • Users use the app many times a day and need a fast way to start or switch to the app.   
  • You have very high requirements on performance.
  • You have high requirements on usability.
  • Your app needs access to device specific hardware and API's such as notifications, calendar, volume controls etc.
  • You are willing to lose some users (unless you develop both html and native apps for iPhone, Android, Windows, Blackberry etc).
  • Your app doesn't need to or should not connect to the Internet.
  • You want to learn Objective-C or other native app programming languages.  
  • Your business model fits well with the monetization model of the App Store, Android market etc. It may be easier to make money off native apps (people tend to pay for apps because it is convenient. It requires close to zero effort to buy an App)   
  • You want yet another channel to be found. (e.g. App store) 
  • You want to use the App in marketing. (Apps are more cool. Few people talk about mobile web apps. People talk about Apps.)
  • You don't want the clutter that comes with a browser (such as the address and navigation bar).
  • Native Apps usually work better or feel more stable when the Internet connection is poor. (E.g. when you are on the subway)
  • You need to hide your code from other developers. (You can't rip html, js and css from a native app)
  • A native app is usually more secure. (Less data traffic. Less chance of phishing attacks and exposing sensitive information)
  • You have content you do not want to be copied or ripped.  
When to choose a HTML Web App
  • You need to push updates for the web app often and promptly (AppStore has a slow process)
  • You don't know who your users are and what devices they will be using.  
  • Your users are coming from random places on the web.
  • You provide links to other websites. (navigation will feel more natural not having to jump in and out of web and native apps)
  • You don't know who is going to maintain the app. (If someone else is going to further develop it there is a bigger change that you find the required competence if you go with HTML) 
  • You already have extensive web development knowledge.
  • You want custom design and branding. (On the web you can make it look however you like)
  • You want complete control of monetisation, including advertising.
  • You want to save time by reusing code between various web apps or projects and by reusing open source code and widgets such as sharing buttons, disqus etc.
  • You want the app to have a long life span. (You don't know what kind of devices and platforms will replace the current. You can be certain that the web will be around for many many years)   
  • You don't want to spend the money (time) to learn and invest in native apps development. 
  • There will be few users. (Little point investing in a new development environment if the App will have only 20 users)
  • A web app can more easily be used by the disabled (unless you develop a specialized native app targeted towards the disabled) 
  • You want better analytics.
  • Faster time to market. You don't have time to wait until you have the capability to develop a native app (it might require learning, developing server integration. A web app can often be made with very little effort and can naturally evolve if you have more time later or find that the web app gets popular).
  • You know that a considerable number of users might not have mobiles that support native apps but might have older web browsers on their phones.  
  • You want to support the development of the free open accessible non-proprietary web.
  • You don't like the political or socio-economic ramifications of walled gardens such as the App store. 
This blog post was written in March 2012. If you are reading this in the future chances are that Web Apps have caught up with Native Apps one many of the their strong points because of HTML5.