TED Pro history API

[pfletch101]

I now have my new TED Pro Home working with my own home energy monitoring and control system, but reprogramming it with what I believe must be a rather early version of the API documentation led to a few challenges!

The initial version of the Polling API I requested and received (PDF Document Version 20130910) contained more than I wanted to know about downloading system data and utility settings, but nothing whatsoever about downloading history information, which is what I needed. I was then sent a Word document (Version 20140120) which contained enough history download API information to get me up and running (the entire history download section of the newer document is in the attached 2-page pdf!), but it did not contain any information about downloading data in Raw format (which you can still do by specifying /history/export.raw as the file name). The documentation also appears to be incorrect in its description of the meaning of the T parameter that you use to specify what sort of data you want to download.

For the benefit of anyone who may also be working with this documentation, the correct specifications for the T parameter appear to be as follows:
0 or 1: Second Data (only about the last hour's data is available)
2: Minute Data (only about the last 48 hours' data is available)
3: Hour Data
4: Day Data
5: Month Data
>=6: Empty file

The data sets returned are also different (mostly in being subsets, but also in different units) than for the corresponding API on the TED 5000. All five sets include MTU identification (now 1-based), date and time, power/energy, and cost as the first four columns (or XML container tags). In addition to these, Second data includes Voltage, and Minute data includes both Voltage and Power Factor. The other sets contain no additional data columns/tags.

A couple of questions remain unanswered:
1) Is the use of raw download mode still recommended for maximum speed and minimum interference with the ECC's normal operations? I am currently downloading the data I need in CSV format, without causing apparent problems, and it seems fairly fast, but XML downloads seem to take quite a while for large data sets.
2) Following on from 1), what is the format of the raw data "packages" for each interval level. I know about Base64 decoding, but the decoded data does not seem to contain the corresponding values in the same order as for the equivalent TED 5000 downloads, and life is too short to treat it as a cryptographic challenge!

[pfletch101]

Just to add an additional challenge for the programmer (!), the definitions of the T parameter are shifted down by 1 if you are getting Spyder data. Spyder channels don't measure or save data as frequently as the main MTUs do, so Second data is not available. The T parameter definitions for Spyder data are:
0 or 1: Minute Data
2: Hour Data
3: Day Data
4: Month Data
>=5: Empty file

Go figure!! The Spyder datasets returned are the same for all time periods: Channel Name, Sample Time, Power (actually Energy), Cost.

[Support7]

crdavis,

Send me an e-mail to support7@theenergydetective.com, and i'll send you a copy of that API.

Best Regards,

TED Forum Support

[crdavis]

Deleted my original post after realizing the documentation was already attached.

I'm curious to get answers to pfletch101's questions. Specifically #2: the format of the raw data coming out of TED PRO.

I get this for the first few rows of CSV second data (looks correct):
MTU, Time, Power, Cost, Voltage
MTU1,06/25/2014 09:16:00,1.312,0.14,119.7
MTU1,06/25/2014 09:15:59,1.312,0.14,119.7
MTU1,06/25/2014 09:15:58,1.312,0.14,119.7
MTU1,06/25/2014 09:15:57,1.308,0.14,119.7
MTU1,06/25/2014 09:15:56,1.316,0.14,119.6

And then the corresponding raw second data (might be offset by a second or 2 from CSV, but should be similar). Printed out each row as a raw string, decoded into base64 and then each byte converted to decimal. I see a 14 in there, which seems to line up with the $0.14 present in the cost column of CSV data.

pJDLqlMgBQAADgAAAK0E4A==
'\xa4\x90\xcb\xaaS \x05\x00\x00\x0e\x00\x00\x00\xad\x04\xe0'
(164, 144, 203, 170, 83, 32, 5, 0, 0, 14, 0, 0, 0, 173, 4, 224)

pI/LqlMgBQAADgAAAK0E3w==
'\xa4\x8f\xcb\xaaS \x05\x00\x00\x0e\x00\x00\x00\xad\x04\xdf'
(164, 143, 203, 170, 83, 32, 5, 0, 0, 14, 0, 0, 0, 173, 4, 223)

pI7LqlMgBQAADgAAAK0E3g==
'\xa4\x8e\xcb\xaaS \x05\x00\x00\x0e\x00\x00\x00\xad\x04\xde'
(164, 142, 203, 170, 83, 32, 5, 0, 0, 14, 0, 0, 0, 173, 4, 222)

[crdavis]

After some playing I came up with this. Hopefully it will help.

Second data (T=0 or T=1) looks like this:
pN/RqlMkBQAADgAAAKgENA==
'\xa4\xdf\xd1\xaaS$\x05\x00\x00\x0e\x00\x00\x00\xa8\x044'
(164, 1403703775, 1316, 14, 1192, 52)

164         #maybe a MTU identifier? it is always 164 for me
1403703775  #taking this as seconds since Jan 1 1970 00:00:00 turns this into UTC time: 2014-06-25 13:42:55
1316        #1.316 W
14          #$0.14
1192        #119.2 V
52          #not sure what this is either; it varies from 0 to 255 and doesn't seem to line up with any visible readings


Minute data (T=2) looks like this (same as second data with PF added after voltage):
pGzRqlMhBQAADgAAAKYEPgP9
'\xa4l\xd1\xaaS!\x05\x00\x00\x0e\x00\x00\x00\xa6\x04>\x03\xfd'
(164, 1403703660, 1313, 14, 1190, 830, 253)
2014-06-25 13:41:00
1.313 W
$0.14
119.0 V
830 #power factor: 83.0%
253 #seems to be the same sort of thing as the final byte in the second data


Hour data (T=3) looks like this (same as minute with voltage and PF removed, power is now in kWh):
pMC5qlOcBAAADAAAAMY=
'\xa4\xc0\xb9\xaaS\x9c\x04\x00\x00\x0c\x00\x00\x00\xc6' 14
(164, 1403697600, 1180, 12, 198)
2014-06-25 12:00:00
1.180 kWh #note the change from W to kWh
$0.12

pLCrqlMpAQAAAwAAACk=
'\xa4\xb0\xab\xaaS)\x01\x00\x00\x03\x00\x00\x00)' 14
(164, 1403694000, 297, 3, 41)
2014-06-25 11:00:00
0.297 kWh #note the change from W to kWh
$0.03

Day data (T=4) looks the same as hour data

Month data (T=5) has 35 bytes of data per row. The first 4 fields are: MTU, time, power (kWh) and cost (cents). I don't have enough history on my TED yet so all of the remaining bytes are 0 for me. I suspect they follow the format shown on p19 here: http://files.theenergydetective.com/docs/TED5000-API-R330.pdf

[crdavis]

Spyder data appears to follow the same format as MTU hour data. All time bases (minute, hour, day, month) have these fields: Unknown Byte (164), Time (4 byte unsigned), Power (4 byte signed), Cost (4 byte signed), Unknown Byte

[Support7]

Hello all,

I should have new documentation with the missing information soon. Thank you for your patience.

Best Regards,

TED Forum Support

[pfletch101]

crdavis:

I confirm most of your findings. You didn't make this 100% clear in your notes (or not to me!), but the voltage and power factor data items are Int16s, while the other items are Int32s. All the power/energy data is passed as Watts/Watt hours, so to get kW/kWh you always divide by 1000. You imply that the scaling changes between the minute and hour data, which I don't find to be the case. My first byte is the same as yours (164) for all data sets that I have looked at, for either of my (2) MTUs.

I do have one month of month data, but the returned raw array is still mostly zero filled (with the exception of the first 11 bytes and bytes 17, 18, and 34). Since most of the data in the TED5000 monthly data set is maxima and minima, which the Pro doesn't seem to record, I suspect that the size of this record is a hangover from the 5000 and will ultimately slim down considerably. The first two items in it appear to be identical to those in the other raw records (Unix date and Energy), but the next is/are either not cost at all or stored/encoded differently. 17/18 may be average voltage. We will hopefully see more when we get the documentation.

[Support7]

Hello all,

There is a new revised Polling API document available. If you would like a copy, please e-mail me at support7@theenergydetective.com and I will send you a copy.

Best Regards,

TED Forum Support

[pfletch101]

One more thing: have you noticed that the date/time returned for each day dataset record represents (once converted) midnight at the end of the relevant day, while the date/time returned for each record in the other datasets represents the start of the interval to which it refers? I think that must represent a bug, since the CSV download for day data shows the correct date, and it may have something to do with conversions to and from local time.

The programmer may want to look at this (hint, hint!)

[pfletch101]

I do have one month of month data, but the returned raw array is still mostly zero filled (with the exception of the first 11 bytes and bytes 17, 18, and 34). Since most of the data in the TED5000 monthly data set is maxima and minima, which the Pro doesn't seem to record, I suspect that the size of this record is a hangover from the 5000 and will ultimately slim down considerably. The first two items in it appear to be identical to those in the other raw records (Unix date and Energy), but the next is/are either not cost at all or stored/encoded differently. 17/18 may be average voltage. We will hopefully see more when we get the documentation.

We do, indeed, see an explanation in the new documentation! All the data beyond byte 8 (counting from 0) in the Raw month data is cost data of one sort or another, but it is split up, so that bytes 9-12 actually only encode the regular baseline-power-dependent component of the monthly cost (in the normal way, as an Int32), and the following bytes encode various other components. 17-20 is the fixed monthly cost component. I'm not sure why it was decided to do it this way.

[pfletch101]

One more thing: have you noticed that the date/time returned for each day dataset record represents (once converted) midnight at the end of the relevant day, while the date/time returned for each record in the other datasets represents the start of the interval to which it refers? I think that must represent a bug, since the CSV download for day data shows the correct date, and it may have something to do with conversions to and from local time.

I have just noticed another, different, weirdness in the Spyder Day CSV data (I haven't checked the Raw Spyder data). The date/time stamp for each record (for me - I suspect it is timezone dependent) includes a time of 19:00 on the day to which the accumulated data 'belongs'. The only way I can make any sense out of this is to suppose that the midnight timestamp is being interpreted as Universal Time and unnecessarily converted to my local time. Subtracting the one hour Daylight Savings time correction from my normal Central US timezone offset would account for the 5-hour difference.

[Everette]

I have just noticed another, different, weirdness in the d-bal Spyder Day CSV data (I haven't checked the Raw Spyder data). The date/time stamp for each record (for me - I suspect it is timezone dependent) includes a time of 19:00 on the day to which the accumulated data 'belongs'.

I noticed this bug too fletch. Any way to fix it for good?

[pfletch101]

Having been working on a TED Pro History Editor for the last few weeks, I now know more than I want to about how Footprints stores and reports timestamps. There is certainly no way of 'fixing' this (if by that you mean forcing the format of the Spyder Day timestamps to be the same as those returned for regular MTUs). TED stores all timestamps as a Unix representation which is based on Universal Time. For the purposes of CSV reporting (only, as far as I know), these are usually converted to local time - it looks as if the code which produces the Spyder Day output simply reports the internal timestamp, without any conversion.

[pfletch101]

An update: it looks as if this bug has been fixed, although I don't know exactly when. I am at ECC and Footprints firmware versions 1.0.717, and I now see that downloaded CSV Spyder Day record timestamps are, in fact, 'pure' dates, which parse to midnight on the specified day.