Monday, 31 March 2014

Dji Naza flyaway air crash investigation

I love my Quadcopter but some weeks ago I started to loose control of the drone multicopter for unexplainable reasons.

Dji has gotten a lot of complaints in the last 6 months from what is often called fly-aways or more humorously called the “Fly to China” feature, (“Fly to home” is a highly regarded safety feature of the Naza M flight controller).

Many of these fly aways are simply pilot errors. There are a lot of pilots out there that should not be flying UAV's in populated areas. Still, reports by experts are plentiful so the problem is real, and serious. A serious accident is bound to happen from a flyaway if the problem is not solved soon.
Many quadcopter forums (such as rcgroupsdjiguys, ausrc, phantompilots, multirotorsforum, fpvlab) have discussed the issue and there are plenty of YouTube videos documenting live flyaways. Even so, Dji seem to be unable to address the issue. They have posted a video with some tips and released a firmware upgrade but the problem remains.

If you have experienced a flyaway please join this survey:

Suggested solutions to the problem can be found all over the place. Here are some solutions found elsewhere (Not my recommendations, just added to show the variety of solution proposals from others):
  • Don’t buy Dji products.
  • Don’t put it in GPS mode.
  • Put tape around the edge of the GPS.
  • Don’t fly when the sun is low in the horizon.
  • Don't fly too close to power lines and cell towers.
  • Reduce vibrations on the bird.
  • Adjust flight mode switch/FS setup (flight mode can jump back and forth due to the control PWM signal being right on the edge).
  • Make sure IOC mode is set up properly and not mistakenly activated/deactivated.
  • Check for loose double side tape on FC or GPS.
  • Replace the GPS/Mag if damaged in crash.
  • Remove FPV Vtx, it may interface with GPS signals.
  • Keep the compass/gps puck and cable away from power leads and battery.
  • Simple shielding on all NAZA inputs to protect data integrity.
  • Remove corrosion and oxidisation of pins and push-connectors.
  • Place Naza FC as close as possible to the center of gravity.
  • Power cycle after compass calibration.
  • Avoid large stick inputs.
  • Recalibrate the compass yet again.
  • Don’t fly when solar activity is high.
  • Make sure the ESCs travel midpoint is at 1520us. Do not use 700us travel midpoint ESC.

As you can see there are several suggestions for how to prevent flyaways, but no one seem to know exactly why it is happening in the first place. It might just be a combination of many things but I would still like to pinpoint the cause of this issue as I have experienced it myself and it is getting both dangerous and expensive.

I am also open for suggestions for how to improve the questionnaire.

My first flyaway (450 quadcopter with gimbal gone haywire) can be seen at the end of this video.

How to gain control if you experience a fly away.

If you have started to experience flyaways you may try to debug the issue by buying the Dji iOSD module to log data.
Another approach is to try systematically changing one thing at a time (environmental condition, copter config, copter parts etc) and see if the fly-aways stop. During this testing create an anchor; a 2 meter long line attached to the ground and tied to the bottom center of your copter.

You may also want to put your name and phone number on your multicopter in case of a flyaway or just not finding it after a crash.

Wednesday, 26 March 2014

Benefits of a schema-less database

A typical argument for choosing a NoSql database (such as a document store or graph database) over a relational database (such as MySql) is that NoSql databases are schemaless and that makes you more agile.

So, what does this actually mean? What are the concrete benefits of being schema-less and how does it actually make you more agile as a developer or as an organization?

In this context we use the term agile not strictly as in Agile software development but more in term of just being faster, more flexible, nimble and adaptable to change.

Here are some of the real world reasons why schemaless makes you more agile:
  • When starting a new development project you don't need to spend the same amount of time on up-front design of the schema (which is likely to change several times anyway). Less need for planning -> more agile.  
  • Once the NoSql is up and running the developer does not need to know about the database. No need to learn SQL or database specific stuff and tools. Less to know about. Less to learn for new developers -> faster development -> more agile.
  • The rigid schema of a relational database (RDBMS) means you have to absolutely follow the schema. It can be harder to push data into the DB as it has to perfectly fit the schema. Being able to add data directly without having to tweak it to match the schema can save you time-> more agile.
  • Minor changes to the model and you will have to change both your code and the schema in the DBMS. No schema -> don't have to make changes in two places -> Less time consuming -> more agile.
  • With a NoSql DB you have fewer ways to pull the data out -> fewer ways to shoot yourself in the foot -> less chance of being bogged down in problems later (e.g. as unforeseen intricate future queries starts to manifest themselves as performance issues) -> more agile.
  • Schema-less -> Less overhead for DB engine -> better performance and scalability -> Less overhead for developers related to scalability -> Fewer devopment obstacles -> more agile. 
  • Changes to the DB schema will often break the solution for other developers and they have to keep updating the DB schema locally even if they are working on something completely unrelated. Syncing DB and code as you pull from your source code repository requires extra work and sometime even extra time trying to figure out why your solution don’t work after Git pull. No schema -> Fewer problems with code/schema not being in sync -> More agile development.
  • ALTER TABLE operations during deployment can be very heavy on the system and will usually result in downtime. Schema changes in general will break the system when code and schema is not in sync during deployment. No schema -> less downtime -> deploy whenever you like-> more agile.
  • Save time when the day comes when you have to denormalize tables because of performance issues. Less overhead related to schema -> more agile.
  • You don't have to deal with artificial link-tables (for many to many relations) because it is built into graph databases. -> fewer things to deal with -> easier to modify code/"schema" -> faster development -> more agile.
  • Eliminates the need for Database administrators or database experts -> fewer people involved and less waiting on experts -> more agile.
  • Less time needed to pick the right datatypes for new database columns -> save time -> more agile.
  • Schema-less -> better DB engine performance -> less chance of having to create separate systems to offload the DB because of heavy queries from big reports etc. -> less complicated system -> more agile. 
  • Save time writing complex SQL joins -> more rapid development -> more agile.
  • Rolling back code doesn't necessarily break the App, you might just not get the data you expect. Easier to deploy/rollback without serious consequences -> more agile.
  • As more and more applications/systems use your RDBMS it is harder to change the schema because of dependencies. Change the schema and you might f* up some other system. With NoSql schema changes other systems might not get the expected data but the chance of runtime errors is less. No schema -> less worry about messing up for other systems -> more agile.
In this blog post I have covered only the benefits of schema-free. There are also drawbacks but for the sake of brevity it is not included here. Being super agile might not be the top prioritization for all organizations.

Wednesday, 5 March 2014

SPARQL and RDF introduction for technical persons

This is a quick introduction to SPARQL and RDF for developers. I believe SPARQL and RDF are such profound innovations that all IT-persons and developers should know about it. So here's me spreading the word.

In this blog post I will show some examples of basic SPARQL queries, but first a short intro to RDF.

Typical relational databases are organized into tables that look something like this:

Table: Persons
Id Name Date of birth Department Id
2 Roy Lachica 22.12.1975 5
4 John Doe 3.4.1973 6
Table: Departments
Id Name
5 Development
6 Human relations

In RDF this would be represented something like this:
subject predicate object 1975-12-22 1973-4-3

This could also be represented as a node graph.
In the example RDF data above we use URI's as identifiers instead of primary key ID columns that are used in relational databases. These URI's and their namespaces are predefined schemas typically defined by other organizations. This is where the big innovation lies.

The problem RDF solves 

In the relational database example we have manually defined the schema our self. There is no way for others to know what the columns mean. Although you might guess from looking at the data. If we were to expose the data through a REST API there is no way for the API consumers to be sure what the data mean, unless they read the documentation. The API consumer will manually have to couple properties, parse and transform data. In short, they will have to make sense of data and its structure.

The innovation

In the triplestore (RDF database) you can put anything in, you can change the data to whatever you like. This will make you more agile. You are not restricted as in a relational database development stack where you would have to change all the above layers if you make a change in the schema.

You also don't have to care about SQL and database engine quirks and performance issues when designing the schema. The extremely simple universal RDF model is a directed graph and you don't need to normalize, denormalize or setup indexes, keys, decide to use stored procedures or not etc. There is no notion of a NULL-value and this will reduce potential bugs.

In RDF, anyone (also computers, not just humans) can make sense of the data, as long as you use known RDF schemas. These schemas are also called vocabularies and ontologies (ontologies are just more advanced vocabularies). There are even some basic semantics inherent in RDF so for simple structures you might not even need to define or reuse a schema.
The URI's tells us what schemas are used and the schema is not hard-coded into the database but instead openly defined. (A schema could also be company internal but that would sort of defeat the purpose of enabling a global database or web of data through connecting disparate data sources)

SPARQL examples

RDF triplestores have SPARQL-endpoints for querying the data. Fortunately university of Mannheim have made such an endpoint openly available. This endpoint exposes a CIA world factbook example database.

The endpoint is located at:
Feel free to check out the endpoint and click around.

This endpoint (with a typical SPARQL user interface) lets you query the triplestore and return the results as JSON, XML or HTML for reading on screen.

You may also type in a query directly in your browser address bar:

Various SPARQL queries

Try copying these into the endpoint SPARQL textfield.

SELECT * WHERE { ?s ?p ?o} limit 10
This will get the first triples (subject-object-predicate set).

SELECT COUNT(*)  { ?s ?p ?o  }
Get number of triples in the database.

SELECT (COUNT(*) AS ?triples_count) { ?s ?p ?o  }
Same as above but with a name for the result, making the output more human readable.

SELECT DISTINCT ?value WHERE { db:Algeria ?value ?o}
Get all properties (the predicate part) of triples where Algeria is part of the triple.

SELECT DISTINCT ?value WHERE { db:Algeria factbook:climate ?value }
Gets the climate in Algeria.

SELECT DISTINCT *WHERE { db:Norway factbook:landboundary ?borderingcountry }
 ORDER BY ?borderingcountry
Get the countries that border to Norway.

SELECT DISTINCT ?country ?literacypercentage
  ?country factbook:literacy_totalpopulation ?literacypercentage .
   FILTER ( ?literacypercentage > 70 )
order by desc(?literacypercentage)
Get all countries with a literacy percentage over 70.

SELECT DISTINCT ?country ?literacypercentage ?populatoncount
  ?country factbook:literacy_totalpopulation ?literacypercentage .
  ?country factbook:population_total  ?populatoncount
   FILTER ( ?literacypercentage > 50 && ?populatoncount>5000000)
order by asc(?literacypercentage)
Get all countries with a literacy percentage over 50 and having a population count of more than 5 million. Sort results by countries with the lowest literacy on top.

SELECT ?country ?population ?growthrate ?capital_city ?area
?x  factbook:name ?country ;
factbook:population_total ?population ;
factbook:populationgrowthrate ?growthrate ;
factbook:capital_name ?capital_city ;
factbook:area_total ?area .
Get country name, population, growth rate, capital city and country area size of all countries.

Further reading

A Relational View of the Semantic Web