Google Timeline Data

I really love the data collected by Google Timeline, and I have fun going through it. But some of the most obvious data aggregates are not available! You cannot get how many miles you’ve biked this year, or how many hours you spent at your office, or how many hours you’ve spent walking, etc. You can see those stats on a day-per-day basis, which I’ll admit is interesting, but I usually already have an instinctual idea of I’ve done on a day-by-day basis. I’d like to imagine that I’d be totally surprised by what my stats would be for a full year.

So I downloaded my raw data from Google Takeout to try and play with, and it turns out to be one humongous geoJSON file. But it seems that it is laid out in the format of each unique timestamp taken, with all the associated data for that point. In most cases, you just get timestamp, latitude/longitude (E7), and accuracy.

{
 "timestampMs" : "1463961031129",
 "latitudeE7" : 377567650,
 "longitudeE7" : -1224178356,
 "accuracy" : 20
 }

But sometimes there’s a list of “activities,” with Google’s “confidence” that the activity listed was the one… being done. They seem to always add up to 100%, if you assume that “on foot” is the same as “walking,” but it isn’t clear to me why both get listed in those cases

{
 "timestampMs" : "1463951764885",
 "latitudeE7" : 377617843,
 "longitudeE7" : -1224204018,
 "accuracy" : 20,
 "activitys" : [ {
 "timestampMs" : "1463951764173",
 "activities" : [ {
 "type" : "onFoot",
 "confidence" : 92
 }, {
 "type" : "walking",
 "confidence" : 92
 }, {
 "type" : "onBicycle",
 "confidence" : 8
 } ]
 } ]
 }

This much is pretty obvious. But then you get activities like “tilting” and other weird things.

Sometimes you get an associated velocity, altitude, and heading. But not usually. Note: I haven’t figured out yet whether the velocity is in kmph or mph.

{
 "timestampMs" : "1463268526270",
 "latitudeE7" : 380429730,
 "longitudeE7" : -1227851373,
 "accuracy" : 10,
 "velocity" : 20,
 "heading" : 239,
 "altitude" : 21,
 "activitys" : [ {
 "timestampMs" : "1463268515541",
 "activities" : [ {
 "type" : "inVehicle",
 "confidence" : 100
 }, {
 "type" : "still",
 "confidence" : 8
 } ],
 "extras" : [ {
 "type" : "value",
 "name" : "vehicle_personal_confidence",
 "intVal" : 100
 } ]
 } ]
 }

So, obviously I’d have to do some calculations based on the latitude and longitude to get data on distances. Which I haven’t done. So I guess I’ll probably just consider the timestamps that have types associated, and for simplicity’s sake I’ll most likely end up only considering the one with the highest “confidence.” With that method, it would be pretty easy to calculate total times. Distances… not as easy, mostly because I haven’t worked with latitudes and longitudes very much. But I’m sure there’s some module out there or something for handling that, so I probably won’t have to learn too much about it 😉

All I’ve done with my data thus far was convert all the timepoints with associated lat/long into human dates with lat/long that are not… E7. So now I have 1,048,575 lines formatted like this

2016-05-22 T14:58:35-07:00    37.7570186, -122.4204018

All the way back to

2013-09-20 T08:18:54-07:00    37.7650828, -122.4172515

Which is the first timepoint I have in my Google Location History.

This format makes it super easy for me to quickly generate maps like

5/22/2016

5/22/2016

5/15/2015

5/15/2015

Or even overlays of multiple days, etc. Which, I realize, is… baaaasically exactly what I can already get from Google Location History through my Timeline.

But! My next step is to calculate how many hours per year were spent doing each activity (walking/biking/driving/etc). There will obviously be far fewer than 1,048,575 timepoints for which there is activity data associated, because most timepoints don’t have any activity.

I haven’t started thinking about the mileage counter yet, so if anybody has any suggestions for handling latitude/longitude points and calculating mileage from those, I guess feel free to let me know. Otherwise, I’ll just let you all know what stats I come up with.

Update 6/5/2016:

My ongoing saga of Google Location History data continues here

About Steen

Steen is a nerdy biologist who spends a lot of time trying to cultivate Chloroflexi. When she isn't working, she likes to draw comics, play video games, and climb.
This entry was posted in JSON, Programming. Bookmark the permalink.

13 Responses to Google Timeline Data

  1. Pingback: Results for week beginning 2016-05-16 | Iron Blogger SF

  2. Pingback: More Google Location Data | Certainly Strange

  3. Josh says:

    How do you convert the latitudeE7 and longitudeE7 to non E7?

    • Steen says:

      You divide it by 10,000,000

      E7 here is just shorthand for the scientific notation of 1×10^7 so that’s why you just move the decimal place 7 spaces and you have it in the more typical coordinates format.

    • Steen says:

      Ah-hah! It figures I should have just checked the documentation. It sounds like it was exactly what I was guessing, but it is good to have it spelled out – and to have a list of all the other “weird ones” and what they mean, too. Thanks!

  4. Dilip says:

    Hello Steen.

    I have a similar predicament. I had downloaded my google timeline data last year in both kml as well as json format. I stopped using that account, but would now like to merge my earlier timeline data with the timeline data of my current account. Is there any way to upload the json or kml file back to be able view the timeline in a readable format? File size is over 34 mb.

    If not, is there any other way or tool available (the way that you have mapped out your biking locations) that can translate this timeline data into any other map readable format (eg – company visited, time in / time out – number of hours spent there, etc.)?

    Do let me know

    • Steen says:

      An interesting idea! As far as personal scripts (like the ones I wrote) for fooling around with the data, yeah it would be pretty easy to merge the two timelines into one. You could read both JSON files into the script and then merge it into a single timeline object and proceed as normally, or you could even just do it manually.

      However, I do not get as sophisticated a readout as Google’s Timeline feature with my own scripts. The JSON object does not explicitly have the shops/locations visited in it, although I imagine you could get that data for yourself by binning the lat/long coordinates and using the Google Maps API for locations of shops etc. It might get kind of hairy at points, though, I haven’t tried that yet. I can’t imagine that Google does this on-the-fly every time, though, so I think they must store that metadata somewhere, but it isn’t accessible to us to download and play with 🙁

      I’ll see if I can come up with something this weekend. I imagine I could at least produce a rudimentary timeline thing – assuming that I can use the Google Maps API to get company info, given lat/long coordinates. I’ll let you know what I come up with!

    • Steen says:

      Awesome, thanks those are good resources for this. And I’m glad you’ve found it helpful!

      Honestly, I should post more about working with location history on here, because I’ve made much more progress since this post. I’ve just been… too lazy to write it up on my personal site I guess? But I think I should make it a priority since people seem to find it helpful!

  5. Trina McLaren says:

    (also btw Steen, I find your avatar strangely disturbing…!)

  6. Tockqua Center says:

    Help! Can anyone tell me if the little red raw data dots on my Google timeline map Can Be edited at all? I don’t want to edit I just want to know if the times stamped and the coordinates associated with the raw data can be edit. Thanks

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.