One of the challenges of marrying sensor data collection and machine learning algorithms to generate usable modes for accomplishing something useful is negotiating the right level of abstraction of data flowing from the sensor to the learner. If the sensor data is too low-level, it functions as little more than noise and the learning algorithm will interpret spurious random patterns as something meaningful. If the data flow is too high-level, you’ve probably wasted time and effort implementing learning infrastructure that is little more than a simple mapping from one high-level concept to another. The trick is finding the right middle ground that maximizes the usefulness of the models being generated with expending as little time and resources processing the data from a low-level noisy signal to something more meaningful.
In Purple Robot, we’ve been dealing with this mismatch between our learners and our location and communication log probes. We know that we’re collecting sufficient raw data that can be used to generate models, but the system isn’t giving us models that work. For example, in some of the apps we’re building for patients with depression, knowing where the patient is helps us understand how they’re spending their time, correlate mood with places they visit, and offer suggestions to help improve their mental state (e.g. “You haven’t left home for the past 18 hours, try taking a walk.”).
Unfortunately, while the mobile device knows where you with an error of less than a couple of meters, it thinks in three-dimensional positions (latitude, longitude, altitude) instead of the more semantic ways humans think about place (home, work, school, etc.). To get around this issue, we’ve implemented automated systems for determining more semantic labels, such as querying the Google Places or Foursquare APIs for labels. We get back more useful information that helps improve our models (“X is in a place that is surrounded by 25% shops, 30% restaurants, 20% parks, and 25% bars”), but this is a generic view of the user’s location that omits any personal meaning or significance given to a place.
Similarly, Purple Robot can access and analyze the local call and SMS logs to extract information that might be useful for determining how a patient’s social contacts influence their behavior and mental state. Purple Robot will try and use any information available in the local address book, but mobile phone users rarely tend to keep their contacts up to date. From Purple Robot’s perspective, it can see that a patient has talked to a particular phone number – which might have a name attached – but from an algorithmic perspective, the values “Jenny, (555) 867-5309” as much meaning as “Z, 4”. This information is useful for distinguishing between individual contacts, but doesn’t shed much more light than that.
After weeks of discussion and brainstorming, we conceded that in order to bump the semantic level of our data up a notch, Purple Robot would have to initiate an interaction with the patient to ask a few targeted questions to assist the sensor moving forward. Since this is not unlike calibrating a measuring instrument, we’ve been calling these interactions “calibrators”.
Calibrators hook into the existing sanity checking framework that alerts the patient if Purple Robot is misconfigured, running out of resources, or fails some other check that indicates the system is performing sub-optimally and a user action can correct the problem. A typical calibrator’s sanity check will ask two questions: if/when was the sensor last calibrated and does the system have enough data to ask good calibration questions. In the case of the communication log sensor, the patient isn’t prompted to calibrate until a sufficient number of communication events (calls, SMSs) are available to label. In the case of the location sensor, the system collects a historical log of places visited, and the patient is invited label the places when the number of logged locations exceeds a couple hundred. Our goal is to only interrupt and query the user as infrequently as possible.
The communication log calibration is pretty simple. After enough calls and texts have been made, Purple Robot prompts the patient to tell us more about the people on the other end of the line. In our case, we’re interested in learning the categories of people, so we present an interface that aggregates the communication log, sorts the contact by frequency of interaction, and the patient can tap the contact’s name/number and select a category from a set of predefined labels (significant other, friend, family, coworker, etc.). The system saves those labels and the next time it transmits communication data to the learner, it will look up the individual’s information from the calibration dictionary and add the category label to the data payload. This allows our learners to not only detect relationships between mental state and individuals (“talking to Coworker Bob makes me more stressed”) but also between mental state and groups (“talking to friends makes me less anxious”).
The location calibrator is a bit more involved. Modern locational technologies return readings that are values on a continuous scale. I can sit perfectly still in a single place and watch my GPS position and it will flutter in the insignificant digits due to variations like weather conditions, satellite position and ambient interference. We can clean this data by rounding readings to a certain level of precision, but the error/flutter in the signal may be larger than you want to round up to. From a user experience perspective, we also don’t want to place a significant labelling burden on the patient to make sense of the data (“label point X. Now label the point 100 meters west of X. Now label the point 100 meters north of X. etc.”).
To get around these two issues, we deploy a technique called “cluster analysis” that can look at a collection of points in a multidimensional space and highlight dense areas where many points are collocated. Algorithms such as k-means are popular approaches to solve this problem, but we decided to use an alternate approach called DBSCAN. In short, the DBSCAN algorithm takes two parameters: what is the minimal distance between points in a cluster and how many points must be close enough for the collection to be considered a cluster. In Purple Robot, DBSCAN allows us to take hundreds of location readings and highlight the ones where the patient spends their time.
Once we generate the location clusters, we have enough information to build an effective user interface that will prompt the patient to label significant places as detected by the DBSCAN algorithm. In our case, we combined a map view with a list view that allows the patient to zoom to a given cluster by clicking the color next to the place label to see where it’s situated on a map, and once they identify the place, they can click the name to assign it a label. The labelled clusters are then saved and when the location sensor generates a new location reading, that reading is tested against the saved clusters to determine if that point would be a member of one of the saved clusters. If it would be a member, then the cluster’s name is appended to the location reading before being sent to the learning algorithm. In our initial testing, this has worked well, but we are still testing the scalability of the approach to determine how much information we can save and process to improve the data we’re generating.
The location and communication log calibrators are only the first of what will probably be many more calibrators. These two sensors were fairly straightforward to tackle with a clear payoff for improved model generation. We are currently discussing taking this approach to other sensors such as implementing calibration for activity detection (walking, running, sitting, etc.) using the motion sensors. In this case, instead of an on-screen user interaction, we may implement an alternative sound-only interface that allows the user to keep their device in its usual location while it instructs them to go through various activities so that the motion calibrator can collect the signals that it needs to train its own internal algorithm for a given patient.
This is an exciting area of development for us, and I look forward to sharing more details and notes as we continue to make progress.