Trello or Jira for dev teams?

Should your development team use Trello or Jira for task management?

Benefits of Trello

• Completely free.
• Less overhead work for managing/administrating the boards/projects/sites.
• Less time consuming for developers. (Less time needed to categorize stuff and update fields)  (Rebuttal: at the expense of structure and reporting)
• Ease of use - A simpler tool like Trello is easier to use for non-technical persons. (Rebuttal: People like stakeholders and others who are not really part of the team should not use the board as it is too technically detailed. Stakeholders should refer to the roadmap and status updates from PM/scrum master)
• View access - Easier to give access to anyone who want to see what we are working on. (Rebuttal: People like stakeholders and others who are not really part of the team should not use the board as it is too detailed. Stakeholders should refer to the roadmap and status updates)
• Create access - Easier to allow external users to post new issues. (Rebuttal: People like stakeholders and others who are not really part of the team should not create items directly for the team.)

Benefits of Jira

• More efficient handling of large backlogs with more analytical triage/selection.
• Better overview and visibility of status and priority across projects (for everyone).
• Avoid loss of information and duplicates as you can more easily find issues with search, filtering, categorization, tagging etc. Also save time finding issues. 
• Better plug-in strategy with more useful plugins.
• More options for integrating with other project planning tools.
• Support for external bug reporting interfaces - Jira has more options for APIs to allow external users to submit bugs and feedback via widgets or other external contact forms etc. and still preserve data quality. 
• Better overview of all my assigned tasks across all projects prioritized in one view.
• Better support for grouping tasks together (related issues. sub issues, epics etc.)
• Compliance - More ISO compliant. E.g. Better audit-logs etc.
• More Agile - Better fit for supporting Scrum and KanBan dev processes. For example Jira supports Epics and Backlogs out of the box among many other things.
• More structure to ensure consistent use across projects (leads to better reports and less confusion/noise).   
• Jira can be run on-premise (in case we in the future want even higher level of security or want to save money)
• Jira has time tracking and estimation support (without having to install quirky plugins)
• Professionalism - A more professional tool will inspire and strengthen the sense of professionalism and developer excellence. 

 

Conclusion

Use Trello for non dev teams, small unstructured ad-hoc teams and short lived projects. Use Jira (or similar tools) if you are professional dev team. 

80 ways to become a 10x developer

The notion of a 10 times more productive developer has long been a controversial topic in the dev-world. Is it really possible for developer A to finish a product in 1/10th of the time developer B would use?

I would say yes if the circumstances are right. Developer productivity, performance or efficiency is a fuzzy concept in the dev world. It is inherently hard to measure productivity because it relies so much on experience, knowledge and creativity. There are infinite ways to complete a piece of software.  For comparison, lets take the job of building a house. In the construction engineering discipline there’s highly established practices for how to construct and assemble the pieces of the house. All the building blocks are standardized and well known. In the dev world we are often free to do whatever we like. We don’t rely on physical items with known tangible properties that need to be put together in a specific way. Instead everything is just lines upon lines of code. Everything is abstract and can be written in infinite ways as long as it compiles.

The 10x developer is a misleading term as there is no agreed upon evaluation criteria and no established way of measuring it. Anyway, in this blog post I’ll list several ways I believe can propel you towards becoming 10x more productive. I am not saying I am a 10x developer but here are some ways I myself and others I have observed have become more productive.

1. Improve your typing skills. Buy the Das Keyboard, a completely blank keyboard with no letters.
2. Learn OS and IDE shortcuts so you don’t have to move the hand over to the mouse.
3. Get an overview and basic understanding of different tools and frameworks out there. This will help you choose the right tool for the job. 
4. Learn about the pros and cons of Open Source and the different licensing models. Learn when to build it yourself and when not to. 
5. Get deep knowledge in architecture, programming languages, algorithms, coding best practices, clean code, design patterns, agile methodologies, XP, unit testing and the other core things related to the art and craft of software development. 
6. Improve your memory. If you can remember details you save time trying to look things up. Do memory exercises, learn recollection techniques etc.
7. Improve your cognition. If your brain can operate at a higher speed you will be able to learn faster and connect the dots faster making your productivity higher. Although you are born with a certain IQ, you can always maximize your cognitive potential. Exercise your brain and stay sharp by always challenging yourself.
8. Be flexible, open-minded and tolerant. If you are, you will be better liked and you’ll have a positive work environment. A good work environment gives you energy and you will boost your productivity. 
9. Learn how to most efficiently use Find and Replace also with regex.
10. Remove repetitive work. An example of repetitive work is writing boilerplate code. Writing boiler place code does not make you a better programmer. You want to do things that serve multiple purposes. Use better tools, make code that writes itself, automate etc.   
11. Work hard and show you have grit. Show that you are willing to get your hands dirty. Don’t shrug off tasks you think will be hard or boring. You do it knowing it has to be done in order to get on with the more fun work. If you constantly complain your way out of tasks that has to be done anyway you just kill productivity. Sacrifice yourself for the team and do the boring task. Over time it will pay off. Show-stoppers or impediments usually escalate. Over time more and more people get invovled in discussing how to deal with it, as time goes there'se more and more noise. All of it could perhaps be avoided if you just sat down at the outset and just fixed the darn issue.
12. Take care of your personal health. Developers who do not take care of their own health and wellbeing are at higher risk of decease. If two years from now you develop diabetes you will for sure be impacted and loose productivity. If you party getting blackout-drunk every week your health will decline and that will impact your ability to stay sharp and learn (see life-long learning).
13. Learn how to analyze stuff. If you can analyze complicated issues you will be better equipped to make better decisions. With better decisions you can avoid going down dead-end roads of development or you will more likely choose a route that will lead you faster to the goal. Learn how to use simple pros and cons lists, learn how to understand complexity. Learn how to decompose issues, view them from different angles, learn about System Dynamics and Systems Thinking.
14. Learn all the code snippets / templates / macros that are available in your IDE.
15. Don’t use plain text editors over fully fledged IDE’s just to show off. (unless you are writing something that is not supported by your IDE)  
16. Build understanding and empathy. If you don’t empathize with your team mates, your boss or your customers you are most likely going to be seen as egocentric, arrogant or distanced, which will create opposition or friction.
17. Memorize language and framework library syntax so you don’t have to google it every time you need to use some feature. 
18. Have stability in your life. With predictability and routines you can better plan your day and devote time for learning. Know your values and stick to them. 
19. Learn to write readable code. Learn all you can about Clean Code, SOLID principles etc.
20. Have focus in your life. Don’t engage in everything that comes to your attention in the private life or at work. If you try to do everything you will be less productive because you waste time on all kinds of things that may not contribute to your long term goals.
21. Always learn. Get obsessed with reading and learning from videos. Coursera, PluralSight, Udemy, YouTube are you friends.
22. Have balance in your life. Don’t work insane hours. Have some fun. Live life. Balance in your life is important to preserve good health. Maintain a sustainable work pace. Knows how to not overwork to avoid and crash and burn.
23. Workout. Maintain a level of physical fitness. Going for a run will clear your mind and you’re head will be ready for more action.  
24. Prioritize and set goals for learning and improving your skills.
25. Learn everywhere. Learn while on the treadmill. Learn by listening to podcasts in your car.
26. Know how to research and make good decisions.
27. Reinforce what you learn by trying it out in practice or by blogging about it, giving talks about it etc. Do Coding Katas or have pet projects where you practice your skills. 
28. Stop wasting time. Of course you are not playing Pokémon Go or watching crap TV. Do you really need to watch season 20 of WhatEverSeries on HBO/Netflix? Don’t watch a movie unless you know it has 7+ stars on IMDB.
29. Invest in developer tools and other software for efficiency. 
30. Invest in hardware such as monitors, a faster computer, a noiseless computer, good prescription glasses, good air-condition, an ergonomic desk and chair.
31. Internalize and reflect on what worked and what didn’t. Don’t just try and fail but understand why things failed. 
32. Spend time on post-incident reviews. Do root cause analysis of failures to learn and prevent the thing from happening again.
33. Be curious and always try to understand how things are connected. This will help you understand how things lead to other things and you’ll get better at deciding what should be fixed right away so it don’t come back later with 10x the negative impact. (see technical debt)
34. Don’t be afraid to ask. Rather than spending days trying to figure out some technical issue on your own, ask a coworker who you think may know. Maybe you just overlooked something. Don’t be afraid to lose face. All in all you’ll excel faster if you let go of the fear to look bad when asking questions.
35. Learn how to write good code. Know how to avoid bugs that will come back to haunt you later and make you spend days trying to debug it.
36. Know what to document and not. Document things that you or the team will get back to many times and will save you time. Don’t document things that only create noise.
37. Spend time on reflection and to come up with the really good ideas. During a typical hectic day at work you may be in a multitasking act and react mode with little time for deep thinking. Find time at a less hectic day or during the weekend for lateral thinking.
38. Stay agile. Be aware of just how much to do e.g. features, documentation, communication etc. not wasting time on things that are not important or deliver value. 
39. Improve your creativity. Learn how the creative process works. Know how to do brainstorming. Learn techniques such as the Six Thinking Hats technique.
40. Dear to try new things. Be open to trying new frameworks etc. (if analysis suggest there is value in using it). Sometimes it pays of a lot to replace some old buggy system with something new.
41. Know how to plan. Good planning goes hand in hand with good decision making. If the plans are bad you’re of course at higher risk of failure. Failures along the way leads to wasted time and less productivity. 
42. Learn how to do requirements specifications. Understand business needs so you don’t start developing something that is not needed or wanted. If you have to scrap a module and start over because the customer rejected it on the grounds it was not what they wanted then you have just wasted a heck lot of time. 
43. In your office room agree not to make phone calls at the desk unless you have to. Always go elsewhere when having a private call so you don’t distract the others.   
44. Build up general knowledge on all aspects relevant to development. If you have some basic knowledge in all relevant areas you are less likely to make bad decisions and you don’t have to rely on others to cover all the things you don’t know you don’t know.
45. Learn to write readable code so you understand the code you wrote 6 months earlier.
46. Learn how to get into the zone.  Find time to shut out everything so you can get into the zone or the flow.  Close down the e-mail reader, social media sites etc. Turn off sounds and vibration on the phone.  
47. Improve your communication skills. With bad communication skills you may miscommunicate which can lead to lots of wasted time. Miscommunications may even result in team misunderstandings and distrust that may lead to long lasting conflicts which of course will be a lot of wasted time and energy.
48. Unsubscribe to e-mail newsletter you rarely read.
49. Share knowledge with your team. Help make the team more effective. If the others are productive people you reduce the chance of being held back by someone who’s unproductive.
50. Only sit near people who you need to communicate with as part of daily work. If you sit near people you don’t work closely with you’ll find yourself distracted when they are on the phone or when someone comes over to their desk discussing things that are irrelevant for you.
51. Agree with your coworkers to always detail out meeting invitations. Have a clear agenda and purpose for a meeting well in advance. If you don’t people will come unprepared to the meeting and many meetings will be a waste.
52. Don’t have meetings unless it’s really needed. Read about good meeting best practices. But don’t take it too far. Meetings can be a good way to let people feel they are heard. If there are no channels for giving input or feedback some people will become dissatisfied.  
53. Know when to outsource, when to hire, when to build or buy. 
54. Be proactive. Sense and foresee things to engage in before it becomes an issue and a big waste of time for everyone.
55. Have an eye on details when programming. Ensure the small things don’t get overlooked.
56. See the big picture. Have an overview and total understanding of the things you are developing and why it’s developed.
57. Get involved in meetups, seminars, conferences etc.
58. Take courses.
59. Take a few breaks during the day to clear your head.
60. Read up on the latest in software engineering best practices on everything from bug tracking, agile methodologies, design patterns, continuous integration, unit testing, integration testing to application monitoring, code reviewing, scalability, cloud hosting, containers, databases, etc. etc.
61. Get deep understanding of the programming languages and frameworks you use.
62. Stay sober with a sharp mind. Limit alcohol intake. No drugs. Eat healthy and get a good night’s sleep. Don’t drink too much coffee too late in the day.
63. Be critical. Don’t take things for granted. Know when to ask critical questions like "What was the reasoning behind choosing this solution?" Asking questions brings new insights. 
64. See challenges in a bigger perspective. Avoid getting completely bogged down in a detailed technical challenge. Rethink, think out of the box and make the problem disappear by seeing it in a completely new light. Instead of asking how can I solve this problem, ask instead is this really the problem, what other things could be solved at the same time if I expanded the scope and introduced more simple solution? 
65. Work on projects that motive you. That usually means choosing to work at a place that can give you challenging and interesting tasks.
66. Don’t delay tasks. Tasks waiting to be done contribute to stress and extra work in the form of overhead. Typically you will regularly have to explain to the team why you’re not doing it and all this noise may even become more time-consuming than the actual time needed to complete the task. 
67. Complete one thing at a time if you can. If you are multitasking and working on many tasks simultaneously during a day you’re not likely going to get into the flow where you can create elegant solutions.
68. Do the things with the most risk first so you can be confident that you’ll succeed with thing you’re working on. If you constantly worry that you may not make the deadline it will be a stress factor that may keep you out of zone.
69. Lean how to do Requirements gathering, making sure that the right questions get asked. Again it is all about making good decisions.
70. Know when to analyze and when to just go and prototype the darn thing. Avoid analysis paralysis but don’t just jump into the task without thinking if there are risks involved, if there are path dependencies etc. 
71. Learn about the business you are in. If you are working as a programmer in bank then learn what you can about banking. Get domain knowledge. Of course you can't learn all there is about banking but sure you can learn about areas within banking that are closest to the system you are working on.
72. Write a ToDo list at the start of the day.
73. When debugging use mock data rather than debugging against a remote API or a big ass database table join to save time waiting for the response.
74. Be part of a great team. A team that stands together and fulfills each other will be stronger and will allow you to become more productive. There’s two ways to become part of a great team, either you switch job/department or you work with what you have and spend more time on recruitment, on knowledge transfer. Be inspiring and lead the way.
75. Be a team player. A team player creates a positive environment that over time creates synergy in that everyone wants to contribute. As everyone on the team is enthusiastic it creates a positive spiral of learning and achievement.
76. Make sure the compilation time, application startup time, debugging cycle time etc. is low so you don’t have to sit and wait doing nothing or even worse, you may be tempted to go on Facebook while you’re waiting. 
77. Set up a proper dev, staging, QA, production environment.
78. Use sticky notes as reminders or as cheat sheets.
79. Close windows and tabs you are not using. They just contribute to noise and make it harder to switch between the tabs/windows you are actually using. They may also distract you and eat CPU/memory. 
80. Understanding the Project Management Diamond/triangle. Learn the tradeoffs between cost, time and scope. Again it all comes down to making good decisions. Now you may say that it’s the manager’s job to make decisions. Eventually you will have to make some decisions yourself and you will also want to have your say in the decisions being made.
81. Be passionate about programming. If you’re not you are not likely to have the motivation and drive to keep learning every day. 
82. Keep a notebook to log your work, scribble down ideas and to take notes. You will need it.
83. Get everyone in the office to agree to only disturb others when there is an urgent matter or the thing to discuss is too complicated, sensitive or risky to be described in an e-mail. Sending an instant message or going over to someone’s desk to ask something will take that person out of the flow. 
84. Don’t listen to music at work. If you need to keep noise out use a noise cancelling headset with no sound. Music may be nice and make the workday more pleasant. My position is that music will lead to distractions. You will have control the audio player, change playlists, share songs with friends etc. It will take away focus when you really need to think super hard about something complicated. If you really want to be 10x then you will have to make sacrifices.
85. Anticipate changes. Always consider feature/change requests that are likely to come. Instead of implementing strictly what the requirements says, evaluate likely changes down the road. If the changes are easy to implement now and would be very difficult to implement later then make all the changes at once.
86. Use naming conventions. All too often I have seen noise and misunderstandings because of unclear, vague or incoherent naming. Stakeholders needs to spent time trying to agree on what to call things. DDD helps in this regard.
87. Understand Statistics and cognitive biases. This is important to be able to judge data and how you and others interpret those data, this again will make you a better decision maker. Poor decisions will take you down roads of wasteful development.

A 10x developer is one who is not just doing things the right way but more importantly doing the right things.

How to rate internal product quality during systems development

There are many ways to define and rate product quality. This product quality rating is based on typical product quality assessments for software products and in particular online software.
This post was written for dev teams who do not have a dedicated QA manager or the like.

Why

So why should you do a product quality rating? It will give you an indication of how well the product is engineered in a more holistic perspective. Many will judge the quality of the product based on what end users say about the product. While end user feedback is a very important indicator of quality or value, the product may still have many issues underneath the covers that are just waiting to blow up and become a big issue for the user or other sides of the business.

This product quality review helps to get a complete overview of the product and its strengths and weaknesses. It helps to plan ahead and reduce risk.

How

The rating can be conducted by the product development team, perhaps in collaboration with others such as internal users and other stakeholders. As several of these categories are only known to the dev-team this rating cannot be done by customers or other external stakeholders.

A score from 0 to 10 can be given where 10 would be the score that the best product on the market would receive in that category.

Extensibility

Is the product considered to be extensible? Can it be enhanced with new capabilities or functionalities without hampering its existing functions?

Availability

Is it product ready for immediate use? Does it require a lot of configuration including technical setup? Is it difficult to get started with the product? Are there long start-up times?

Features

Features are the “bells and whistles” of products. How complete is the product? Compared to competitors how would you rate the feature-set in terms of user value? Are you missing important features? Are there many half-finished features?

Performance

Refers to throughput and latency. Can the system handle the user load? Is the system perceived as slow at times?

Accessibility

Is the system accessible for impaired/handicapped users?

Reliability

Does the product perform flawlessly under stated conditions for a long period of time?

Correctness / Conformance

Is it free from faults/bugs? Is it aligned with specifications? Does it conform to standards?

Efficiency

Does the product perform effectively without wasting resources?

Maintainability

Is it easy to maintain code?

Understandability

Is it easy to understand and grasp what the product is for? Does it seem too complex?

Usability

Is it easy to use and learn?

Supportability

Is it well documented? Is it easy to support?

Scalability

Does the product scale well and can it handle controlled increased load? Is it efficient when new resources are added?

Robustness

Is the product able to withstand harsh conditions and rough handling? It should be durable enough to endure erratic variations in stress as well as pressure that too with minimal damage or alteration to its functionality. Does it handle bad or corrupt data?

Security

Are there any likelihood of potential security breaches due to poor coding practices and architecture etc? Have you done penetration test or security audits? Do you have logging, exception handling etc.

Elegance

Is the product stylish, good looking, giving a good first impression?

Further reading

http://sloanreview.mit.edu/article/what-does-product-quality-really-mean/

Why testing tasks should be part of the task board when you don't have experienced testers

Many Scrum teams feel there's something not quite right about testing and their use of a task board. In this blog post I'll go in-depth on the issue of having testing-related tasks or not on the task board. I have found that this is a typical question especially in beginner Scrum teams.

Example Scrum team scenario

To keep this blog post relatively short I will focus on the following scenario:

1. Automated testing code coverage is medium to low so there is an extra need for manual testing.
2. The product is relatively complicated with lots of intricate scenarios/settings to test.
3. The product is a SaaS with thousands of users.
4. You do not have dedicated testers but you have plenty of access to people outside the team that can help to carry out testing. 
5. Testers are not experienced in testing. They know the product to be tested but are not super users. They prefer not to use developer oriented testing tools because they are not highly technical persons. 
6. Testers outside the team come and go and they are quite busy so they need a simple way to carry out the tests without having to log in to a complicated testing suite. Because of the turnover you do not want to provide a lot of training for testers. 
7. You do not have a test manager, QA lead etc.
8. Developers are also inexperienced testers and have limited knowledge about QA.
9. Developers are writing the test scripts because the testers are not qualified to write high quality test cases. 

This scenario may describe a team with a medium Agile maturity level so other more fundamental actions could also be needed such as QA coaching, adding QA related metrics etc. but we will focus on the question of testing tasks on the board.

What is testing anyway?

You should never rollout a feature/User Story/Product Backlog Item (PBI) without testing. Someone has to test it. This might take the form of unit testing, acceptance testing, security/penetration testing, exploratory testing, regression testing, performance testing, load testing, code review, integration testing, web tests e.g. Selenium. In addition to local testing during development. Maybe you want to add UX user testing and maybe you even need to do some regulatory testing.

The purpose of the task board

Before we discuss if testing tasks should be on the board we should first do a recap of what a task board is for. The task board, as the name implies, is for tasks. The purpose of the task board is to keep track of tasks and create visibility to the team and other stakeholders. This helps the team make sure the right things are completed at the right time [4]. The three pillars of the Empirical process which Scrum and Agile is based upon is Visibility, Inspection and Adaption. Tasks on the task board enables visibility and transparency.

Now, some teams try to cram more information onto the board by adding lanes for server environments or lanes such as "Ready for testing", "In testing", "Ready for staging" etc.
The starter task board with the 3 lanes: "To do", "In progress/WIP/Doing" and "Done" is still the recommended basic setup [1],[2],[3],[4],[5],[6],[9]. Remember that the task board is primarily for tracking tasks.

A task can be in progress, done etc. To say a task is in QA/staging or in testing does not make sense. A user story/feature can be in testing but a task is a piece of work to be done. You don't test a testing task. This inconsistency lies at the root of disagreements on how to visualize testing on the board.

Why not have explicit testing tasks on the board

Typical arguments for not having testing tasks for each story:

• It's extra work to add test-related tasks to each user story. Each user story will often have the same duplicated testing tasks. Seems like unnecessary extra work to add them for each story. 
• Testing is an integral part of development so we don't need tasks for it. It is part of the programming work.  
• More tasks means there will be more overhead needed to pass all the tasks through the lanes of the task board.
• The board gets messy when we add tasks that are to be partially carried out by others not in the core team. We like to have full control of what all the tasks are and not have things that relate to other persons in there.
• Testing is something that is done after we have deployed to the QA servers. Seems out of place and not logical to have them on the board. Testing is sort of another phase of the process and it seems illogical to have it alongside programming tasks. 

Some of these arguments include faulty assumptions.

• "The board gets messy when we add tasks that are to be carried out by others not in the core team". This claim assumes that the team don't need to do anything related to testing. Someone has to write the test cases, make sure the test environment is testable, someone has to coordinate testing, make sure testing gets done, provide support for testers etc. There are also tests that only developers can do like performance testing.
• "Testing is sort of another phase". According to Agile practices this is just wrong [7],[8],[10],[11]. This thinking leads some teams to add a testing lane to their board which again strengthens a view that testing is a phase in the sprint and so test-related tasks are not needed on the board.
  

  

  Example task board with lane for testing

  This can also have the effect that epics are not broken down into small enough user stories and tasks. Since it does not always make sense to have a task in testing you may end up having tasks that are actually user stories. When a task on the board is actually a user story it makes sense to have the "task" In testing. 

Testing is not a phase, but a way of life
   - Elisabeth Hendrickson

Ideally testing should be a collaborative effort between developers and testers going on in parallel [8],[10],[13],[14]. In our scenario this is difficult since there are no skilled testers so developers have to step up and write test cases, provide testing support and coordinate testing [11].

In Scrum and Agile development we strive to complete one by one user story. Testing is of course part of that. One by one story is completed to reduce risk and to be able to deliver value increments.

Alternatives to testing tasks on the task board

Based on our scenario I have identified a few alternatives to having testing tasks.
  

Alternative	Implication for test script writing	Implication on carrying out the tests
Developers write test specifications at the start of the sprint based on the requirements specifications, dev tasks, DOD and Acceptance Criteria.	You don't yet have a working feature so you will not be able to try it out and come up with all the test cases at the start. There will be changes underway so the test script will be outdated. To avoid rewriting the test cases or have to come back to it at a later time the developer will delay writing the test cases. Eventually the developer may forget to do it.	Poor test coverage and outdated test cases that cannot be carried out. Rush at the end of the sprint to get the test script together.
Developers update the test specification continuously as they go along.	Will be forgotten because the developer is in a coding mode and the team does not yet have a high quality focus.	The developer may forget to do testing tasks such as performance tests, integration tests, code reviews, security/penetration tests etc. Rush at the end of the sprint to get the test script together.
Developers write the tests when all dev tasks of a story are done.	Who is responsible? Several developers have worked on the user story. It is not clear who should write the test cases so it doesn't get done.	Rush at the end of the sprint to get the test script together.
Developers write test cases for each dev task.	Many tasks are not testable [12]. Often there are no relevant manual test cases to write and there are different types of testing needed of different tasks and so the developer would constantly need to think about testing. Not all developers care that much about quality. Also the test script needs to be organized into a readable non-overlapping set of test cases. You may not end up with a complete and easy to use test script just from piecing together tests from the individual dev tasks. There is no reminder or test task to check off as completed so the developer may forget about it or defer writing the test cases.	Missing test cases. Tests that are hard to understand. Overlapping tests. Rush at the end of the sprint to get the test script together. If you have a lane for testing this does not mean that tests will automatically be carried out. Because some tasks are testable and others not you don't get into a systematic process of writing test cases and eventually things are forgotten or delayed. Also there might not be a clear mapping between the dev task and its test cases so others will not know if the test cases are written or not.
Developers write the tests at day x into the sprint.	The developer will always want to complete as much features as possible to look good or because he don't like writing test specs. Quality is jeopardized because the developer rush to complete the test specification in order to be able to complete the testing before the sprint ends. This is not Agile.	Testing is postponed, it gets chunked up towards the end of the sprint. You may not be able to complete any stories at all because testing was halted for some reason. The feedback cycle lags behind so critical bugs found during testing could not be fixed in time before the sprint ended or the team have to work overtime, again.
Developers test by trying to break each other's code and find bugs.	In this case there is less need to write a test script. Although this may sound fun to developers this alternative ultimately depend on the QA maturity of the developers and the process. If developers are poor in testing they will find fewer bugs.	Without specific tasks with a timeslot developers may do a sloppy testing job especially if there are no guidelines from managers on expected quality levels or if there are no other incentives for finding bugs.

Some readers might have noticed how central the test script has become in our scenario. Maybe you would like to suggest dropping test scripts all together and just end the discussion right here. Developers should be end to end developers and have the skills needed to test everything along the way as until the story is done [14]. In our scenario with a complicated product this is simply not an option because a test script is absolutely needed for QA to make sure edge cases are both identified and tested. Without a test script you get into a testing procrastination situation. You don't know where to even start testing and what has been tested. Systematic testing is needed in our scenario.

Typical problems in immature Scrum teams when not having testing tasks for each story

Developers usually don't like to do or prepare testing other than technical unit testing and performance tests so you may end up with poor tests and test scripts being ready too late. Developers love to code and develop stuff and often find test-related work boring.

Typically the team will end up postponing testing as a bulk job to be done towards the end of the sprint [7]. As you don't have explicit tasks for writing test specs and the task of writing them is usually seen as boring work for developers they will tend to focus on development and postpone writing test scripts as long as they can. This means that they will usually start to work on another story before the first one has gone through testing. You are now in a situation where developers try to have all stories ready to be tested some time before the sprint ends so testers can take over and begin their work. This is what we would call mini-waterfall (not Agile).

Another problem is responsibility. Who is responsible for writing the manual test scripts, the acceptance tests etc? Who is responsible for coordinating or doing the actual testing. If there's no assigned person you can bet it will be deferred or not done at all when the QA maturity level is low amongst developers.

Another side-effect of not having testing tasks on the board is that some person will voluntarily or not take the role of writing the test specifications because that person know it is the only way it will be done. This person might be good at it and he might even like to do it. The problem here is that it creates a dependency. What happens if this person goes away on vacation or is busy doing other stuff for a period of time? The testing grinds to a halt or you have a severe drop in test coverage.

Why have testing tasks as part of the task board

Example of Scrum board with testing tasks

Better estimates

Time will go into writing test specs, so estimates for it should be registered somewhere. Time will be spent setting up the test environment, doing initial integration tests, regression tests etc. Time will go into fixing the bugs found during testing.

By using test-related tasks you can during Sprint planning have a more conscious discussion around what kind of testing is needed, how much time will be needed and how much extra time should be added for fixing bugs found during testing). With explicit testing tasks you will be able to plan better and come up with better estimates. 

Increased Quality

Sometimes tests can only be done by developers. E.g. performance testing or testing that require specific tools e.g. testing infrastructure related stuff. By defining these tests as explicit you don't forget to do them and you have accountability. Everyone sees who is responsible and if it has been done or not.

If you add test-related tasks during sprint planning you will have a slot to think about testing. You will be able to kick-off some thoughts about what needs to be tested, how thoroughly etc.

If there's no defined task for writing the test script you might be pressured to hurry the job of writing the test cases because the burn-down chart will look bad if you spend too much time writing it. When there are no testing-related tasks on the board there are no estimates to burn. The developer might feel he look less productive than other developers who are working on defined dev tasks so the developer may want to just get the test cases done as quick as possible. Not thought through test cases = poor testing.

If you have testing-related tasks on the board you will be reminded if there are some other types of testing missing.

Who is responsible for updating the test scripts when a bug is found and there was no test to cover it? Without a person in charge of overseeing the testing of a user story this may very well be forgotten.

Reporting and status visibility

If you don't have a task for testing the user story you can't display the status of the testing work on the board. There is no way to see if the testing has started and how many hours remain etc. Testing-related work as any other task can have impediments. If you have test-related work on the board it's easier to have discussions about status and impediments.

Efficiency

In our scenario someone in the team has to coordinate or connect developers with testers. Testers will have questions for how to carry out some test task. There are usually questions such as; is it a bug or is it a known issue? A developer has to be available to answer questions such as why a test task is not testable. The correct conditions, for example the right data might not be present to be able to carry out the test. Without a explicit task for the responsibility of coordinating and supporting the testers the tests might very well stop without anyone knowing. You might have unclear communication lines and get extra noise in the team because people don't know who is responsible for overseeing the testing.

Avoid duplicated work. If the developers test each other work it is still useful to have testing tasks so you know which developer is testing who's work.

Continuous improvement

Reducing waste is central to the Agile movement. If everything is visible you get a better overview of everything going on and how things depend on other things. With testing tasks you can more easily improve the entire process.

Without tasks and responsibilities you don't have accountability. If the test script is bad and there are bugs because of poor tests we know who wrote the test spec and we can improve the process by talking to the responsible persons.

References

[1] http://www.clemensreijnen.nl/post/2012/05/24/09-Getting-Testing-Done-in-the-Sprint-e28093-Test-Tasks-on-the-Board.aspx
[2] http://www.mountaingoatsoftware.com/agile/scrum/task-boards
[3] https://msdn.microsoft.com/en-us/library/jj161049.aspx
[4] http://leankit.com/project-management/task-boards/

[5] http://blogs.atlassian.com/2012/08/greenhopper-bonfire-agile-development-testing-awesomeness/

[6] http://manifesto.co.uk/agile-concepts-scrum-task-board/

[7] http://www.ambysoft.com/essays/agileTesting.html
[8] https://www.youtube.com/watch?v=bqrOnIECCSg
[9] http://www.slideshare.net/mikedep01/building-scrum-boards-that-radiate-information-generic
[10] http://smartbear.com/all-resources/articles/what-is-agile-testing/
[11] https://www.atlassian.com/software-testing
[12] http://programmers.stackexchange.com/a/113837
[13] http://www.scaledagileframework.com/developers-and-testers/
[14] https://www.youtube.com/watch?v=yRP29wFqu20

Why fixed length sprints are important in Scrum

Have you been on a scrum team that sometimes change the length of a sprint or postpones a sprint? You are not alone, and in this blog post I list reasons why doing this is a bad idea. 

Oh, and in case you wondered; fixed sprint length IS a well-documented best practice of Scrum [1, 2, 3, 4, 7, 8, 9, 10].

Sprints best have consistent durations.... A new Sprint starts immediately after the conclusion of the previous Sprint [1].

Some common reasons why a team would want to alter the sprint cadence now and then:

• Vacations.
• Important deadlines/releases.
• Unforeseen issues taking up a lot of time of the sprint.
• Now able to complete important user-stories to be released. 
• Staff unavailable (sick team members etc.) 

Here is the full list of why you should stick to a fixed length sprints (fixed cadence):

1. Improve coordination with the rest of the organization 

It will be easier for everyone to know what kind of Sprint phase the team is in (e.g. planning, development, testing, reviewing). This predictability allows for better coordination and planning throughout the organization. The entire organization can self-organize around this rhythm. 

It also helps you to steer expectations. With a fixed length sprint everyone (PO, management, customers, users, and other stakeholders) will know when they can expect new updates. They will know that a new updates typically comes every 3 weeks (or whatever your sprint length is).

2. Save time planning and booking meetings

With a fixed sprint length you can schedule and make plans more easily. A sprint is always the same length, that's it. Now you can schedule Scrum events for the next year ahead and you can do it in one go. You will have less overhead in planning because you don't have to negotiate what should be the sprint length each time, you don't have to make complicated man-days calculations and you don't have to change all future sprints because you changed the length of the current sprint.

Since you can schedule meetings in advance, people can plan their appointments around the Sprint events, rather than sprints being tweaked to fit the calendar of key team members or stakeholders. 

3. Better quality

In general a fixed-length sprint will remove variability and this will lead to more predictability, stability, better flow, better and more sustainable work environment, less chaos. This of course leads to higher quality. 

If you allow sprints to be shortened, say for example because the top management really want to have the next release earlier you not only loose rhythm but you will either have to work overtime, reduce quality, add more resources, or scope down the current sprint. If you scope down you waste resources and throw away planning/analysis that is already done and other work that has been started. If you decide not to scope down you are typically left with one option on such a short notice. You end up reducing quality for example by pressuring the testing to complete faster or by not letting developers do refactoring and other chores. If you decide to work overtime you still jeopardize quality since tired or pressured team members make more mistakes. 

4. More effective work with better flow

With fixed length sprints it is much easier to get into a flow of doing all the Scrum related events such as backlog grooming, retrospectives, sprint reviews. With a predictable schedule you don't have to think about these events. When is the next meeting, will there even be one?

Without this fixed start and stop you may be tempted to move the start and stop date. You might even delay starting the next sprint long after the previous one ended. Beginner Scrum teams in this situation might not schedule the sprints for the months ahead but instead just start a new sprint when it makes sense. Before you know it you skip a retrospective and have to do sprint planning in the middle of a sprint. All this leads to more unpredictability, questions, chaos and less flow which eventually results in lower performance of the team.

5. Improve estimates

Empiricism [10]. If the sprint length varies from sprint to sprint, then the amount of work accomplished in each sprint tells you next to nothing about what to expect in future sprints. If you know there are many vacation days coming up in the next sprint you are better off continuing with the regular sprint length/cadence, be it with fewer actual work days and just disregard the burndown or dev velocity of that sprint.  

6. Enable continuous improvement 

Sprints can more easily be compared. In Scrum we try to fix as many parameters as possible; Time, Quality, Resources are fixed but we change the work from sprint to sprint. By reducing the complexity down to just one adjustable parameter (backlog items) it will be easier to have discussions around opportunities for improving the process. The more variables you change the more complex the dynamics of the process will be and it will be much more difficult to compare sprints to improve the process. If you change multiple parameters around each sprint, how are you going to know for sure what actually contributed to things going better or worse?

7. Improve ability to predict when future features will be released

The organization or customer will know how long it will take to get new releases out the door if suddenly a new feature absolutely has to be introduced as soon as possible. E.g. If the team is in the middle of a sprint and the sprint length is 3 weeks it will take about 5 weeks to see the newly requested feature in production. If the sprint length is variable it would be far less obvious when you could expect the new feature.

For products planned several sprints down the road it would be even more unpredictable without a fixed length iteration.

Velocity (the rate at which a team delivers stories from the product backlog) is useful in forecasting when capabilities can be deployed and available to end users. Without fixed-length sprints, any measure of “work accomplished” is useless in forecasting. Alternatively you will have to make adjustments in the calculations of velocity based on the sprint length. These sorts of calculations become a waste of time as they are hard to get exact and just add extra work to the Scrum master. You can never hope to be 100% precise on task estimates so Velocity will never be exact science anyway. By having fixed length sprints you reduce one source or error in calculating velocity. Keep it simple, stay lean and pragmatic. Aim for predictability and simplicity and forecasting will be easier and less time consuming.

Sure there will always be holidays, sick days, etc. If a sprint has less resources you simply commit to less [5]. You calculate how many man hours is available and plan the scope accordingly. You may now say that this will affect velocity and number of storypoints delivered for these sprints will be lower. Yes and that is fine because you know the reason why. In the Velocity-over-time-graph you can just make a small note about vacation etc. when velocity had a dip. If you know most of the sprint is going to be all messed up because too many people are away you can also decide for the team to do something completely different. You could let developers take the sprint off doing courses, prototypes etc.

8. Improve team motivation

Fixed length sprint helps the team to deliver continuously and provides a sense of accomplishment at the end of every sprint. The team also commit to a chunk of work at the start of the sprint. The team gets motivated to deliver as they themselves committed to doing the work.

With fixed length sprints you have predictability and the team feels more at ease. People like to have routines and familiarity [5]. Fixed length sprints contribute to discipline, habits and traditions that creates team identity. "This is our team and this is how we do it." This again strengthens cohesion and team spirit, which is important for highly effective teams.

If Scrum events falls on random days of the week every sprint then it is hard to build good habits.

9. Accountability

Fixed length sprints with reviews at the end of the sprint encourages the team to deliver on the sprint review. Without a fixed sprint calendar the chance of skipping reviews gets much bigger. Without this regular review delivery the team may procrastinate, get caught up in coding what is fun rather than sticking to the plan or get distracted by other things. Developers don't want to look bad so they will deliver when it is demo time.

10. Better planning of backlog and roadmap

PO can better prioritize and split user stories now that meaningful data on velocity can be accumulated from sprint to sprint. PO can use this velocity as a guide to plan upcoming sprints.

11. Essential for multi team synchronization and alignment

When you scale to multiple dev teams doing Scrum (co-located or not) contributing to the same solution or product, it is obvious that synchronizing the teams will be more effective [6].

If the different teams operate with different Sprint lengths it goes without saying that the teams will not be in sync.

Conclusion

Scrum is all about delivering software in regular intervals to enable fast feedback and to get a effective cadence for the team. Variable Sprint lengths may seem like a good idea, especially in the short term, but in the long run it will be devastating to the progress of a team. There will always be sprints with  holidays and sickness but the show must go on. Do not postpone sprints or adjust the length of the sprint, just keep moving and avoid unnecessary chaos by accepting that some sprints will have lower velocity.  

If you liked this blog post, you may also want to read Scrum Best Practices: Choosing the Right Sprint Length.

References

[1] http://www.scrumguides.org/docs/scrumguide/v1/Scrum-Guide-US.pdf
[2] http://www.infoq.com/minibooks/scrum-xp-from-the-trenches

[3] http://www.agilitysw.com/blog/31-scrum-whys-fixed-length-sprints
[4] http://books.google.no/books?id=3vGEcOfCkdwC (page 67)  

[5] http://whitewaterprojects.com/2010/12/07/2-models-to-address-the-holiday-sprints/

[6] http://scaledagileframework.com/sprint-execution/
[7] http://www.agileadvice.com/2013/05/15/referenceinformation/the-rules-of-scrum-every-sprint-is-the-same-length
[8] http://blogs.telerik.com/agile-blog/posts/13-09-11/the-importance-of-timeboxing-and-iterations-for-agile-planning
[9] https://www.scrumalliance.org/community/articles/2012/january/a-sprint-is-not-a-mini-waterfall

[10] https://artisanagility.com/never-extend-a-sprint/ 

Benefits of a schema-less database

A typical argument for choosing a NoSql database (such as a document store or graph database) over a relational database (such as MySql) is that NoSql databases are schemaless and that makes you more agile.

So, what does this actually mean? What are the concrete benefits of being schema-less and how does it actually make you more agile as a developer or as an organization?

In this context we use the term agile not strictly as in Agile software development but more in term of just being faster, more flexible, nimble and adaptable to change.

Here are some of the real world reasons why schemaless makes you more agile:

• When starting a new development project you don't need to spend the same amount of time on up-front design of the schema (which is likely to change several times anyway). Less need for planning -> more agile.  
• Once the NoSql is up and running the developer does not need to know about the database. No need to learn SQL or database specific stuff and tools. Less to know about. Less to learn for new developers -> faster development -> more agile.
• The rigid schema of a relational database (RDBMS) means you have to absolutely follow the schema. It can be harder to push data into the DB as it has to perfectly fit the schema. Being able to add data directly without having to tweak it to match the schema can save you time-> more agile.
• Minor changes to the model and you will have to change both your code and the schema in the DBMS. No schema -> don't have to make changes in two places -> Less time consuming -> more agile.
• With a NoSql DB you have fewer ways to pull the data out -> fewer ways to shoot yourself in the foot -> less chance of being bogged down in problems later (e.g. as unforeseen intricate future queries starts to manifest themselves as performance issues) -> more agile.
• Schema-less -> Less overhead for DB engine -> better performance and scalability -> Less overhead for developers related to scalability -> Fewer devopment obstacles -> more agile. 
• Changes to the DB schema will often break the solution for other developers and they have to keep updating the DB schema locally even if they are working on something completely unrelated. Syncing DB and code as you pull from your source code repository requires extra work and sometime even extra time trying to figure out why your solution don’t work after Git pull. No schema -> Fewer problems with code/schema not being in sync -> More agile development.
• ALTER TABLE operations during deployment can be very heavy on the system and will usually result in downtime. Schema changes in general will break the system when code and schema is not in sync during deployment. No schema -> less downtime -> deploy whenever you like-> more agile.
• Save time when the day comes when you have to denormalize tables because of performance issues. Less overhead related to schema -> more agile.
• You don't have to deal with artificial link-tables (for many to many relations) because it is built into graph databases. -> fewer things to deal with -> easier to modify code/"schema" -> faster development -> more agile.
• Eliminates the need for Database administrators or database experts -> fewer people involved and less waiting on experts -> more agile.
• Less time needed to pick the right datatypes for new database columns -> save time -> more agile.
• Schema-less -> better DB engine performance -> less chance of having to create separate systems to offload the DB because of heavy queries from big reports etc. -> less complicated system -> more agile. 
• Save time writing complex SQL joins -> more rapid development -> more agile.
• Rolling back code doesn't necessarily break the App, you might just not get the data you expect. Easier to deploy/rollback without serious consequences -> more agile.
• As more and more applications/systems use your RDBMS it is harder to change the schema because of dependencies. Change the schema and you might f* up some other system. With NoSql schema changes other systems might not get the expected data but the chance of runtime errors is less. No schema -> less worry about messing up for other systems -> more agile.

In this blog post I have covered only the benefits of schema-free. There are also drawbacks but for the sake of brevity it is not included here. Being super agile might not be the top prioritization for all organizations.

50 reasons why I don't like SharePoint development

I was introduced to SharePoint in 2001 when it was called SharePoint Portal Server 2001, codename Tahoe. From time to time I have been put on SharePoint projects but I never came to like it.

I think I have worked on more than 10 SharePoint projects, some of them fairly big. The last time I used SharePoint was back in 2011 with SharePoint 2010.

This blog post is not intended as an evaluation of SharePoint with a socio-technical economic corporate or whatever perspective. SharePoint might, or might not, be a good choice for your development project. This post is Not about SharePoint as a viable product. This post is solely about what I think about SharePoint as a development platform.

Please note that I am talking from my experience as a developer who happen to like simplicity, structure and who cares about usability. What you will find here is simply my personal view of SharePoint development. This is based on doing deep custom SharePoint development, not just configuring out-of-the-box SharePoint sites. Also note that this blog post was written before the release of SharePoint 2013.

Not just me being an old crank
Even prominent people within the .Net community such as Carl Franklin and Richard Campbell openly say what they think of SharePoint. In .Net rocks podcast episode 734 Richard Campbell makes the following joke: "The thing about SharePoint is you start on having a problem you think you can solve with SharePoint and then you have two problems."

Richard Campbell even goes as far as to say SharePoint makes you cry.

You'll even find developers of SharePoint at Microsoft out right saying: "It is not, and never will be, a viable developer platform."

The 50 reasons why I don't like SharePoint development and why I'll hopefully never do any more of it:

1. SharePoint kills creativity with its configurations. Flicking on xml/xsl files is not very inspiring.
2. The backend user interface is mostly not very intuitive. I as a developer don't like to develop systems that suck.
3. SharePoint as a framework is too big and complex and you never get to feel that you master it all.
4. The product is pricy and too big so I can't use it for my private projects. That means I will not be able to reuse the skills on my hobby projects.
5. SharePoint experience has limited transfer value for other type of development projects. 
6. There is usually IT politics involved since it's a big decision to start building the competence and infrastructure. With politics often follow dirty games, power struggles etc.
7. SharePoint requires lots of training and long-lasting changes in the behaviour of users. This means that you don't get to see the results of your work and how users actually end up using the system. By the time the use of the solution has stabilized you are long gone on other clients, other projects or have left the firm.
8. Once an organization and the development team has acquired SharePoint licences and competency they tend to reuse SharePoint on other IT projects where SharePoint is not the best fit. SharePoint often becomes overkill and results in very large development costs. You as the developer may end up having developed a system that is doomed to fail. 
9. Organizations who have acquired SharePoint for their internal systems often want to reuse the platform for their public facing website to consolidate their infrastructure/licenses. It brings with it a large overhead and the public facing website becomes too complex to manage , maintain and evolve. You as a developer might get stuck doing boring support or very difficult maintenence.
10. Agile methods can become problematic. SharePoint content types cannot easily be changed once it's in production or even in testing environments. This means projects have to be fairly waterfallish.
11. Difficult to write unit tests. Poor separation of concerns.
12. SharePoint version 2010 does not support .Net framework 4.0.
13. The platform is resource demanding and becomes slow on your dev machine, especially if you have a VM setup.
14. Debugging is slow. Timeout occurs. When attaching the debugging you'll often hook on the wrong w3wp.exe.
15. Poor HTML rendered. Takes LOTS of time to customize rendering.
16. No Asp.Net MVC support (unless you are only using the SharePoint API) .
17. Heavy frontend html/js/css that runs slowly in the browser.
18. Lots of XSL files. I never liked XSLT.
19. Making control adapters is a headache, must deploy but then reactivate web app feature to deploy app_browsers file. compat.browser with control adapters debugging often don't work.
20. The ghosted/unghosted feature of content types is a big headache.
21. Some super users will have the rights to change contenttypes, delete site columns, delete or move lists. This is dangerous if you have custom backend code that uses these lists.
22. Very difficult to investigate content in the database when debugging.
23. Often you do not need workflow, content approval and document versioning etc. All these things make the solution more complex and you will have to remove or deactivate these things if you don't need them. Scripting removal/deactivation takes time.
24. Lots of things happening under the hood that is not documented very well. New developers, not familiar with SharePoint will often make bad choices that will be very hard to fix later.
25. SharePoint requires a lot of competency so you often have persons specializing i various SharePoint fields and often you'll have parts of the IT department that do SharePoint maintenance/operations tasks. This creates dependencies between people and you often get stuck waiting for other persons or departments to deploy or do other development tasks. If the required person or department is unavailable you are of course stuck and you look bad.
26. Memory leaks are easy to add (although there are tools to find them), App pools regularly crash.
27. Packaging and deployment is of course different from anything else. Additional competencies needed.
28. The sheer magnitude of the product is difficult for developers. You'll need different third party tools. All the knowledge required is overwhelming.
29. When the site goes down or suddenly become slow after an upgrade it is often very difficult to pin-point the error.
30. The log file (ULS) is always filled with a ton of entries and you have no idea what half of them are. The ULS log is typically 2000 lines long and the error messages are very bad.
31. The API returns data that are concatenated and needs to be parsed.
32. Has to run on Windows server. It can run on Windows 7 but not really straight forward to configure your local environment if your team has a good continuous integration setup.
33. Unpredictable system performance. When you have performance problems and investigate you will often not find the problem because there are just too many moving parts. Suddenly the performance is better. You look at automatic database tuning, at changes in content and changes settings/configurations. You look for memory leaks, do performance testing in the testing environment, restore different versions of the production database, you look at visitor statistics, web server logs, sql logs, application event logs, ULS logs, look at timer jobs, various manual jobs that can have been started, you look at users behaviour during the period, you check automated response testing like Gomez, what's hitting your site, crawlers, what content is being read, when is backup and indexing on the database running. Has the MOSS search crawler been running etc but the problem is often a mix of interrelated issues so it's hard to find by investigating sub systems one after the other.
34. SharePoint has too many configuration options. An example is regional settings on each site. You might get problems showing dates consistently and not understand why.
35. Lots of access control and post release configuration required. You'll have to determine what content types, site templates, page layouts, webparts should be allowed to use by who and where.
36. If you on a page want to hide fields that are empty or add some conditional formatting you'll have to create custom webparts or usercontrols. Webparts that should be hidden in special scenarios requires additional development.
37. Some out-of-the-box webparts (such as search) creates poor html which makes them harder to style and look a like on different browsers.
38. Since everything is customizable in SharePoint you end up with less user friendly dialogs. For example if you use a very simple page publishing workflow there are still going to be superfluous user interface elements and interactions that makes it messy for the user.
39. Ambiguous error messages.
40. In a script-it-all dev environment you often configure some part of the SharePoint solution and to test it you do a deploy of the entire site by building it from scratch or running some upgrade scripts. You find that the configuration you just wrote did not work for some reason. The scripts takes 20 minutes to run so you spend the rest of the day tweeking the configuration files to get it to work by running the deployment scripts over and over.
41. Setting up a sitedefinition always seems to be a problem. Settings up a new sitedefinition and configuring what features, page layouts, content types, master pages, welcome page should be used for a site seems like a straight forward job. But this for some reason always fails and you have to modify the configuration and redeploy and redeploy and redeploy until you get it right.
42. A single typo in a setup config file can destroy your site and will be hard to locate.
43. You are using a very large platform and you can't just copy one installation to a new site and start from there.
44. In most cases you will start by reconfiguring the site from scratch. So most of what you do is to write code or configurations to build and deploy your site instead of spending time developing cool features using cool code.
45. It is difficult to script everything. There are simply to many customizations and configurations that can be done. There is always some manual change that someone forgot to script.
46. Too many layers of development. In other projects you might just have to copy the entire site, make some adjustments in a few config files, then set up the database and you are good to go. But with SharePoint you will have to write code that builds the things that builds your solution (unless you are doing simple projects using the SharePoint designer).
47. The whole development process tends to get messy because things are just hard and complicated with SharePoint.
48. Creating Internet facing websites are difficult because the platform has a lot of overhead. Lots of effort to go around things, removing things and making the site more accessible and faster.
49. Because SharePoint development is so complex things very often takes more time than estimated so you miss deadlines or have to work overtime.
50. If you have a SharePoint project with high expectations on visual design, usability or accessibility then you will require designers with good knowledge of SharePoint. This can be hard to come by and/or will require excellent collaboration between developers and designers. Often challenging.
51. Microsoft usually makes big changes to SharePoint with each new version. If you have not followed best practices you are in for lots of work if you are upgrading in the future. What the best practices are takes time to learn.
52. The preferred way to store data in SharePoint is with SharePoint lists. If you do not use lists and instead go with separate databases you can end up with a fragmented solution with many databases (SharePoint warranty breach if you meddle with the internal SharePoint databases). If you use SharePoint lists you are likely to run in to issues with the abstraction layers and problems trying to put relational data in lists.

Further reading

http://www.cmswire.com/cms/information-management/the-art-of-sharepoint-success-10-reasons-why-sharepoint-projects-fail-012592.php
http://codebetter.com/jefferypalermo/2007/09/13/sharepoint-is-not-a-good-development-platform

SharePoint or Asp.Net MVC for building a new and simple web site

I was recently on a project where we were faced with the choice of what platform to base a scoped down prototype web site on. Based on the team competencies and the fact that we had already developed several SharePoint sites for the customer we were basically down to two choices, SharePoint or developing an Asp.Net MVC site from scratch. I consider Asp.Net MVC superior to Asp.Net so a regular Asp.Net WebForms project was out of the question. SharePoint is mainly a collaboration solution best suited for Intranets and portals. Even so, many companies use the platform for their company website because of the growing amount of SharePoint developers and other benefits of the platform such as extensibility.

The purpose of this blog post is to share knowledge for others who might be faced with the same choice of using SharePoint or Asp.Net MVC for a very small web site with few requirements.

About the team
We were 4 developers, a scrum master, a part time interaction designer and a part time graphical designer on the team. None of the team members were SharePoint or Asp.Net MVC specialists. None had any experience worth mentioning on SharePoint 2010. The average experience in years with programming of the team members were less than 10 years. Most had worked with SharePoint for some years but none of the team members had in-depth expert knowledge on all sides of SharePoint development.

A short note about the scope of the project
Time span of the project was about 2 months including planning, design, development and testing. Without going to much into details, here are the features of the site:

• Integration of some external social media widgets.
• Very basic publishing of simple content.
• Branding and fancy visual design.
• Some commenting, mailing list sign up and feedback features.

The site was a pilot project so we did not have strict requirements on quality assurance etc.

Platform choice
The decision was to use SharePoint as most of the developers were more familiar with SharePoint over Asp.Net MVC. The decision was inline with the company IT-strategy and most of the developers were keen on trying out the new SharePoint 2010.

Experiences
At the time of the project it just happened that I was starting a new hobby project using Asp.Net MVC on my spare time so I had the opportunity to compare SharePoint and Asp.Net MVC development side by side. The two projects were similar in many ways. In the comparison below I only discuss issues that are relevant to both platforms. Many of these problems have also been encountered in previous projects so they might be said to be typical problems you might expect to run into.

Problems we encountered while starting a new SharePoint 2010 project

• We had to switch to a new development environment and install Windows 7 since SP2010 requires Windows 7. Installing Windows 7, setting up SharePoint and configuring all your nice to have software typically takes two days.
• We later got licensing problems for the OS and Visual Studio. This is not a big problem but all of these small non-code related things combined make a lot of noise.
• SP Feature mess: We had trouble setting up page layouts. It kept getting ghosted automatically. We found out that some meta tags attributes on the page layout caused it to automatically getting ghosted during deployment. This resulted in several wasted dev hours.
• SharePoint has a lot of built in JS such as the init.js file. One of the social media widgets we wanted to use would not run along side the SharePoint JS on IE7. Debugging minified widget code and thousands of SharePoint lines was not anything we wanted to do so we had to write some custom code. Again several hours wasted on debugging and custom development.
• SharePoint 2010 still renders bad html and lots of other page elements, js and other markup that is not needed for a Internet facing site. The site even tried loading Silverlight when used as an Internet facing site. Chrome crashed because of the Silverlight plugin. This was fixed by adding a SPSecurityTrimmedControl around the ribbonrow.
• We added several web parts to the web part gallery and we encountered problems that made the add-webpart ribbon unusable since the add button got hidden beneath the ribbon bottom edge on some browsers.
• We encountered problems with the NoteField. We wanted more than 255 characters for the text. When deciding to use the html rich text field instead we had to change the content type and this is not a walk in the park if you have already created some test pages you would like to keep.
• Each time a new content type field is added there's often problems for the other developers. The other developers need to upgrade their solution.
• VS caching bug. While doing deploy from SharePoint within VS we sometimes got this error: deploying Activating feature 'OurFeatureName' ...
• Error occurred in deployment step 'Activate Features': The field with Id {e3575ee4-1b6e-43fc-b243-77f727f33e16} defined in feature {a36d44c1-b523-4235-950a-59ed763d2588} was found in the current site collection or in a subsite. This error had to do with some state being cached by Visual Studio 2010. In this case we had to restart VS.
• SharePoint includes a bunch of js and css files. To optimize page rendering on the client we cleaned up and customized the minimal.master masterpage using editmode controls etc.
• We created XSL for the Content by Query (CBQ) web parts and had to figure out why the CBQ didn't list our pages. Debugging this was time consuming.
• We also had problems with web parts not being deployed because of inconsistencies between .dwp and .webpart files.
• Another problem related to web parts not being deployed had to do with mismatch in the SP2010 .spdef files that might be caused by code merging problems.
• A typical error many do on SharePoint. XSL files packaged incorrectly and end user in anonymous mode don't have access to the xsl.-files and the page breaks.
• Then we had problem with the server. The system was a small pilot project so the server we got was an old outdated server. SharePoint 2010 simply did not run on the old server without extensive reconfiguration.
• We copied a SharePoint feature from another project. We forgot to change a GUID and things just did not work. Debugging, again not trivial.
• SharePoint 2010 does not support IE6 so we had to add a custom warning message for IE6 users. The lack of IE6-support has in retrospective been a problem because it also resulted in down prioritizing IE6 for the front end anonymous view of the site and we have got several complaints from end users who did not understand why the site looked bad on their browser. Also showing of the site is also problematic because the company still used IE6 as the default browser so this lead to embarrassment as we didn't always have control of the installed browser at the location where the web site was presented.
• The page layout or list we tried to add at some point simply would not be added to SharePoint. No easy way to figure out what was wrong with the xml. The schema.xml for a list is not particularly pretty and inviting to work with. There are of course tools to help you with this but the work all becomes very complex and awkward unless this is the only thing you do for a living.
• Then, one of us had problems updating the web.config. Updates did not affect the site on the developer’s machine. After an hour wasted work we found out that SharePoint 2010 use a different web.config file. The new web.config file is under the layouts folder in the 14 hive.
• Next problem; SharePoint redirected mobile devices to a mobile site by default so we had to customize this also.
• Then we got problems on the Publishing Images doc lib. Some unexpected error occurred when navigating to it. Still don't remember what was done to fix it. Simply too many problems and solutions to keep track of.
• We had a lot of static html code that were inserted in content editor web parts. Part of this static html was some javascripts used for client side paging and other GUI effects. In SharePoint this had the affect that the JavaScript altered the content of the web part while the user was editing the web part. When saving the web part the user would be saving the jQuery altered version of the html. To fix this we had to link to a separate file instead and update this one rather than editing the web part content itself. With Asp.Net MVC this would be a simple text field with html. The wysiwyg of SharePoint just made things more time consuming here.
• As with other typical SharePoint projects we had to set up content types, set default page layouts for a page library, what page layouts are available for a site etc. All these things take time to configure in the site definition. This is typically tedious and boring work.

Problems I encountered when starting a new Asp.Net MVC project

• First problem: You code everything from scratch this can be a frightening enough it self. Using some sample code as a starting point might help a lot.
• You get 'Object reference not set to an instance of an object' errors and you can't figure out what is wrong. Asp.Net MVC uses a lot of convention over configuration so there's a lot happening under the hood. Debugging this might be difficult because there a lot of abstractions and inner workings that you might not want to dive into.
• More convention over configuration problems. If you miss a convention or have a convention implemented wrong you will often not get a compile time error message and it’s difficult to locate the error.
• Figuring out how the routing works on XP with IIS5.1 is difficult. Not very intuitive if you are still on XP.

Conclusion
Asp.Net MVC is simply a web application framework. SharePoint is a software platform for collaboration and web publishing and has a rich set of capabilities out of the box. So basically what you are doing in SharePoint development is to customize the existing features, tweaking features, overriding stuff, extending features or even stripping them way. On Asp.Net MVC, on the other hand, you develop the site basically from scratch. For most developers it will be more interesting and exciting to develop something from scratch that they have full control over, not having to abide by the complexities of SharePoint and having to sit and configure XML files all day long.

If you know that you are going to use some built in feature of SharePoint you should consider SharePoint, but if you are not going to use any of the strong selling points of SharePoint such as document management, workflow, compliance, publishing, document search etc you should definitely build it your self.

Benefits of Asp.Net MVC over SharePoint for small and simple web sites

• Asp.Net MVC uses convention over configuration. In most cases this makes development very fast as the conventions reduces the work to be done as well as providing a model (MVC) for architecting and for how to arrange code in a consistent and clean manner that further reduce complexity and speeds up development. Faster development means cheaper development and enables early testing which is important for a prototype.
• You feel more productive as everything you do has a purpose and is directly affecting the solution. Every line of code has direct impact on the actual feature you are working on. SharePoint xml configuration is often used to configure some feature that you don't even need in the solution but you have to configure it anyway just because it's there and available be default to users. A lot of tasks on SharePoint simply seem like a waste of time since you are not working on the required features but instead on removing the features that are not required.
• Most developers find it more fun to write code than writing XML. When writing code you get to be creative and show how good or smart you are. With XML configurations you have to stick to the schema, no room for creativity.
• By using Asp.Net MVC we would have Azure as an alternative or backup hosting model.
• SharePoint development requires a lot of competency. To customize a SharePoint site you need to know web development as well as XSL, the SharePoint object model, feature stapling, site definitions, WSP packaging, SharePoint security model and a bunch of other things. Since very few developers are experts in all of these areas it takes a lot of coordination and teamwork. More dependencies means more development time.
• A part from the social aspects and skills required there are the complexities of the technical solution it self. In general, SharePoint introduces a lot of complexity that makes development less fast. Lots of different type of code in different places, lots of classes, lots of different configurations on different levels, various tools such as STSADM, lots of scripting commands to remember, many folder locations. Lots of concepts and features. If you want to look into the database to try figure out what SharePoint is doing you'll also meet a wall of complexity there also. More complexity equals developer headache.

Benefits of SharePoint over Asp.Net MVC for small and simple web sites

• SharePoint has a lot of built in features, some of these might be exactly what you need. For example, Web parts that can be moved between web part zones was one such feature we made use of.
• As with all IT-projects you must take into consideration future requirements. SharePoint's comprehensive platform might be what you need for your site in the future when your site grows larger and features such as versioning, document management etc may be required.

The big take-away from this blog post

Complexity = more development time.

Requirements reminder

In smaller dev teams in some organizations, small projects often start without much consideration to requirements other than some design sketches. This is a starting point that can lead to several issues down the line, and it is typical when there are no solution architects involved. Once the team (or single developer) has chosen a technical path it might be very expensive to redo the system. The longer the system has been in use by end users the harder it is to rearchitect it.

In this blog post I list typical areas of IT systems. The purpose of this list is to trigger deliberation that can help you formulate a simple requirements specification. 

• Access (Who should have access)
• Accessibility, WAI
• Accuracy
• Adaptability
• Audit and control
• Availability (Service level agreement, downtime)
• Backup
• Branding
• Browser support, clients
• Capacity
• Certification
• Compatibility
• Compliance
• Completeness
• Configuration management
• Configure-ability (can it be reconfigured)
• Correctness (no bugs)
• Cultural and political
• Currency (data is up to date)
• Data structures
• Dependency on other parties
• Dependability
• Deployment (deployment procedures, life cycle turn around and velocity)
• Design (and layout)
• Disaster recovery
• Documentation
• Editorial, publishing
• Efficiency (resource consumption IT, energy, human, performance vs effort)
• Emotional factors (fun, absorbing)
• Environmental tolerance
• Escrow
• Error tolerance (wrong user input)
• Error messages. (how and where)
• Ethical considerations
• Extensibility (adding features, upgrades)
• Failure management
• Failure tolerance (defect in system execution)
• Install-ability (easy to install?)
• Internationalization (e.g., different countries, languages)
• Interoperability (properly interfaced with and working together with something else)
• Language (style of writing, target group)
• Learnability (easy to learn)
• Legal (licensing issues, patents-infringement-avoidability)
• Logging
• Logistics
• Maintainability
• Memorability (easy to remember how to do thing in the system)
• Mobility (Can it be used on the move)
• Modifiability
• Multi user scenarios (collaboration, access to the same objects at the same time etc).
• Network topology
• Open source
• Operational , manual procedures
• Operability (possible to perform tasks in accordance with operations manual?)
• Packaging
• Performance / response time (performance engineering)
• Personalization (personal user experience)
• Performance (Jitter, Response time, Latency)
• Personell (and resources)
• Platform compatibility
• Portability
• Precision of data
• Price
• Privacy (GDPR)
• Provisions
• Protocols
• Quality (faults delivered or fixed)
• Reuse-ability
• Recovery (e.g. mean time to recovery - MTTR)
• Reliability (e.g. mean time between failures - MTBF)
• Reporting
• Resilience
• Resource constraints (processor speed, memory, disk space, network bandwidth)
• Response time
• Reliability (operates without failure under specified use)
• Robustness (also functions under abnormal conditions)
• Roles (Who can add, edit, delete, view)
• Safety (preventing / dealing with accidental harm)
• Scalability (horizontal, vertical)
• Schedule-ability
• Scope
• Security (tolerance against malicious harm)
• Source code
• Stability
• Subset-ability (easy to make different functionality subsets)
• Support (end user support)
• Survivability (essential services continue in spite of accidental or malicious harm)
• Style
• Templates
• Testability
• Throughput
• Traceability
• Training
• Transportability (possible to move system)
• Unambiguity
• Usability (ease of use)
• Usefulness (useful to use)
• User experience (system perceived as fun, satisfying, enjoyable, entertaining, helpful, motivating, aesthetically pleasing, emotionally fulfilling, rewarding and supportive of creativity etc)
• Validation (of user inputs)
• Verifiability
• Variability (degree to which different variants exist)
• Withdraw-ability (can a specific version be withdrawn and replaced with a previous, version)
• Workflow
• Workload (low perceived workload by user)

References

http://en.wikipedia.org/wiki/Non-functional_requirement

http://www.idi.ntnu.no/grupper/su/fordypningsprosjekt-2005/eide-fordyp05.pdf