Forums

2.5 digit bcd to 8 bit encoding?

Started by N_Cook November 18, 2014
On Tue, 18 Nov 2014 11:18:06 -0800, David Platt wrote:

> In article <m4faqm$i3e$1@dont-email.me>, N_Cook <diverse@tcp.co.uk> > wrote: > >>Don't know if it is compressing or encoding but using some of the easily >>resolvable "illegal" ABCDEF codes of the otherwise unused bits in BCD to >>encode the units data and then use the pc to extract and manipulate back >>to 0 to 3.99. > > If you're truly trying to send 400 different codes, using a single 8-bit > byte (which has only 255 possible values) for each reading, and you need > to be able to handle each and every set of readings which could ever > occur in sequence... I believe it cannot be done. The "pigeonhole > principle" forbids it. > > In practice, if the readings aren't changing too quickly or too > arbitrarily, you may be able to use compression of some sort to get the > data rate down to an average of one (or less) 8-bit byte transferred, > per reading taken. This will be an average... there will be times when > you need to transmit two bytes to handle a reading... and there will be > some amount of additional latency added by the compression scheme. > > The gotcha is that no matter what lossless-compression scheme you > choose, there will *always* be *some* patterns of inputs which don't > compress, or which expand slightly. You're trying to pack about 8.5 > bits worth of information into each 8-bit byte, and that cannot always > be done.
Somewhere in the hash of his original posting he mentions that the data is changing slowly. If you used one bit for a flag, you could send 128 different values of absolute data to initialize, then send a delta in the range -64 +63 (if you used two's compliment) at every sample that doesn't happen to fall on a legal "absolute" value. If your "absolute" data ranged from 0.00 to 5.08 in steps of 0.04, then as long as the data really did move slowly, you'd never be off by more than 0.02, and then only for one transmission time. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com
In article <StydneML0JXVAvbJnZ2dnUU7-KednZ2d@giganews.com>,
Tim Wescott  <seemywebsite@myfooter.really> wrote:

>Somewhere in the hash of his original posting he mentions that the data is >changing slowly. > >If you used one bit for a flag, you could send 128 different values of >absolute data to initialize, then send a delta in the range -64 +63 (if >you used two's compliment) at every sample that doesn't happen to fall on >a legal "absolute" value. > >If your "absolute" data ranged from 0.00 to 5.08 in steps of 0.04, then as >long as the data really did move slowly, you'd never be off by more than >0.02, and then only for one transmission time.
That would be akin to an ADPCM (adaptive delta PCM) encoding, and yes, it could work reasonably well. If you're strictly limited to sending only one byte per sample time, it would be "lossy compression" due to the slight inaccuracies it would develop from time to time, but it might well be "good enough for folk music".
On Tue, 18 Nov 2014 13:47:21 -0800, David Platt wrote:

> In article <StydneML0JXVAvbJnZ2dnUU7-KednZ2d@giganews.com>, > Tim Wescott <seemywebsite@myfooter.really> wrote: > >>Somewhere in the hash of his original posting he mentions that the data >>is changing slowly. >> >>If you used one bit for a flag, you could send 128 different values of >>absolute data to initialize, then send a delta in the range -64 +63 (if >>you used two's compliment) at every sample that doesn't happen to fall >>on a legal "absolute" value. >> >>If your "absolute" data ranged from 0.00 to 5.08 in steps of 0.04, then >>as long as the data really did move slowly, you'd never be off by more >>than 0.02, and then only for one transmission time. > > That would be akin to an ADPCM (adaptive delta PCM) encoding, and yes, > it could work reasonably well. If you're strictly limited to sending > only one byte per sample time, it would be "lossy compression" due to > the slight inaccuracies it would develop from time to time, but it might > well be "good enough for folk music".
I think it'd be much smarter to just send the reading, in ASCII, with a linefeed at the end to boot -- that way you plug the module into a serial port and look at the result on a terminal when you're debugging problems. But the OP's prose is somewhat impenetrable. -- Tim Wescott Wescott Design Services http://www.wescottdesign.com
Tim Wescott wrote:
> On Tue, 18 Nov 2014 11:18:06 -0800, David Platt wrote: > >> In article <m4faqm$i3e$1@dont-email.me>, N_Cook <diverse@tcp.co.uk> >> wrote: >> >>> Don't know if it is compressing or encoding but using some of the easily >>> resolvable "illegal" ABCDEF codes of the otherwise unused bits in BCD to >>> encode the units data and then use the pc to extract and manipulate back >>> to 0 to 3.99. >> >> If you're truly trying to send 400 different codes, using a single 8-bit >> byte (which has only 255 possible values) for each reading, and you need >> to be able to handle each and every set of readings which could ever >> occur in sequence... I believe it cannot be done. The "pigeonhole >> principle" forbids it. >> >> In practice, if the readings aren't changing too quickly or too >> arbitrarily, you may be able to use compression of some sort to get the >> data rate down to an average of one (or less) 8-bit byte transferred, >> per reading taken. This will be an average... there will be times when >> you need to transmit two bytes to handle a reading... and there will be >> some amount of additional latency added by the compression scheme. >> >> The gotcha is that no matter what lossless-compression scheme you >> choose, there will *always* be *some* patterns of inputs which don't >> compress, or which expand slightly. You're trying to pack about 8.5 >> bits worth of information into each 8-bit byte, and that cannot always >> be done. > > Somewhere in the hash of his original posting he mentions that the data is > changing slowly. > > If you used one bit for a flag, you could send 128 different values of > absolute data to initialize, then send a delta in the range -64 +63 (if > you used two's compliment) at every sample that doesn't happen to fall on > a legal "absolute" value. >
So basically ADPCM rather than PCM.
> If your "absolute" data ranged from 0.00 to 5.08 in steps of 0.04, then as > long as the data really did move slowly, you'd never be off by more than > 0.02, and then only for one transmission time. >
In cases, slew rate limiting can improve absolute performance. -- Les Cargill
On 11/18/2014 3:30 AM, N_Cook wrote:
> So 0.00 to 3.99 to 8bit and then via simple 8bit RS232. The data is well > behaved , just going up or down slowly. > As limited to hardware conversion/encoding at the sensor end (pc at the other > so no decoding limitation there), thinking > Send as packed BCD of the tenths and hundredths and then hardware encode onto > MSB only when it is 1001 "9", the units 0to 3 reading as 2bit binary on the D6 > and D7 bits. Then on the pc "stick or twist" the units logged according to > whether the next tenth sent reading is 8 or 0. > I suppose in the general case 9.99 could be encoded like this for 0 to 9.99 > using D1 and D2 on the LSB but would not be reliable for general data or > jittering around the .9. > Any other harware conversion/encoding idea I might consider?
(sigh) Complete sentences and *thoughts* would make this *much* easier to understand... "Send as packed BCD of the tenths and hundredths" OK, so there goes your first byte (packed BCD=2*4b) Now, you've got two more bits to encode, somewhere. Presumably, you don't want to spend (transfer) another byte. So, all you can do is encode the bit in "time". I.e., build some sense of state into your algorithm (neglecting, for the moment, the startup issues of such an algorithm -- maybe your signal ALWAYS starts from some known value, etc). So, if you've seen something approaching the high end of your encoding range (8x, 9x, etc.), you can send subsequent values that EXCEED that as 2r101xxxxx - 2r11111111 (or, 2r11111001, etc.). I.e., 16rA0-16rA9 could represent 1.00-1.09, 16rB0-16rB9 as 1.10-1.19, etc. And, as you reach another "threshold", the encoding once again changes -- until you have extended your range to include 3.90-3.99. This really seems kludgey. All to save 1 bit in a transmission? (0-3.99 can be encoded in 9 bits IF you want 0.01 precision in your values; less if you are willing to settle for less precision!) A more worthwhile approach (if you are wed to BCD and can't cope with binary) might be to Chen-Ho encode your 2.25 digits into 8b.
On 11/18/2014 9:16 AM, N_Cook wrote:
> The way I see it , there is 100 datapoints , in 2 ranks tenths and hundredths ,
Correct.
> leaving 156 unused places to hide 4 datapoints for the units.
No. If you use one of these datapoints, the other 100 go away because you are NOT using one of the 100 datapoints that represents them! Pick one of the letters in the following string and insert it between these two X's: X X ABCDEFGHIJKLMNOPQRSTUVWXYZ Now, pick one of the digits in the following string and insert it between those same two X's! 0123456789 Gee, where did your letter choice go??
> Using the most > appropriate place to hide them , which for hardware manipulation would seem to > be D5 and D6 , unused in 8 and 9 representations of BCD,1000 and 1001, of which > 9 is the easiest to gate from.
You have more data available than space to fit (store) it. You have to encode the data in some "out of band" manner -- e.g., time/state. But, that will only work if you can cleanly and deterministically *get* to the "correct" state in some fixed amount of time. E.g., what do you do if you turn on your device (or, the PC) and the value to be sent is 3.49? As you can only represent 0.00-0.99 (or 0.00-2.55 or <whatever>!), the first value that you ship to the PC will be "wrong". E.g., you may send (the code for) 0.99 -- but, clearly the value is NOT 0.99! You may then quickly follow up with some special code knowing the PC now has "memory of" 0.99 and, effectively, slew to a higher value (e.g., perhaps 0xFF tells the PC "add 1.00 to the previous value that you saw" while anything between 0xA0 and 0xE9 says "add the value LESS 0xA0 to the most recent value that you have"). So, send 0x99. Then, 0xFF (PC now is thinking 0.99+1=1.99). Then, another 0xFF (1.99+1=2.99) then 0xE9 (2.99+0xE9-0xA0=3.48 -- if I've done my hex math correctly) then 0xA1 to arrive at 3.48+0.01=3.49. But, each of these incremental values was "wrong" -- the input was 3.49 for ALL of them (and, presumably, hasnt changed in the interim)! [Of course this encoding won't work. It's just something trivial to show you what the problem is. You are relying on time/state to convey additional information. So, there is always a risk that you will not be in-sync with the data unless you can guarantee that it never slews faster than your code can accommodate. Startup is obviously a bad time for such protocols!]
On 11/18/2014 10:18 PM, Don Y wrote:
> On 11/18/2014 9:16 AM, N_Cook wrote: >> The way I see it , there is 100 datapoints , in 2 ranks tenths and hundredths , > > Correct. > >> leaving 156 unused places to hide 4 datapoints for the units. > > No. If you use one of these datapoints, the other 100 go away because you > are NOT using one of the 100 datapoints that represents them! > > Pick one of the letters in the following string and insert it between > these two X's: X X > > ABCDEFGHIJKLMNOPQRSTUVWXYZ > > Now, pick one of the digits in the following string and insert it between > those same two X's! > > 0123456789 > > Gee, where did your letter choice go??
Perhaps I wasn't explicit enough in this explanation. The letters represent your "100 code points". The digits represent the 156 *unused* code points. The space between the two X's is the spot where you put *one* value that you select from the 256 code points available -- 100 used and 156 unused. You can't fit two choices in space for one!
On 11/18/2014 6:06 PM, Tim Wescott wrote:
> On Tue, 18 Nov 2014 13:47:21 -0800, David Platt wrote: > >> In article <StydneML0JXVAvbJnZ2dnUU7-KednZ2d@giganews.com>, >> Tim Wescott <seemywebsite@myfooter.really> wrote: >> >>> Somewhere in the hash of his original posting he mentions that the data >>> is changing slowly. >>> >>> If you used one bit for a flag, you could send 128 different values of >>> absolute data to initialize, then send a delta in the range -64 +63 (if >>> you used two's compliment) at every sample that doesn't happen to fall >>> on a legal "absolute" value. >>> >>> If your "absolute" data ranged from 0.00 to 5.08 in steps of 0.04, then >>> as long as the data really did move slowly, you'd never be off by more >>> than 0.02, and then only for one transmission time. >> >> That would be akin to an ADPCM (adaptive delta PCM) encoding, and yes, >> it could work reasonably well. If you're strictly limited to sending >> only one byte per sample time, it would be "lossy compression" due to >> the slight inaccuracies it would develop from time to time, but it might >> well be "good enough for folk music". > > I think it'd be much smarter to just send the reading, in ASCII, with a > linefeed at the end to boot -- that way you plug the module into a serial > port and look at the result on a terminal when you're debugging problems. > > But the OP's prose is somewhat impenetrable.
Actually I think I cracked the code. He is thinking that he can utilize some of the "don't care" bits that occur in the two digits, '8' and '9'. Not fully realizing that these can indeed be used... if these digits are transmitted. But when the two lsds are characters other than an 8 or 9 there are no don't care bits so you lose that capacity. -- Rick
On 18/11/2014 19:16, Phil Hobbs wrote:
> On 11/18/2014 01:25 PM, Tim Wescott wrote: >> On Tue, 18 Nov 2014 11:33:53 +0000, N_Cook wrote: >> >>> On 18/11/2014 10:51, rickman wrote: >>>> On 11/18/2014 5:30 AM, N_Cook wrote: >>>>> So 0.00 to 3.99 to 8bit and then via simple 8bit RS232. The data is >>>>> well behaved , just going up or down slowly. >>>>> As limited to hardware conversion/encoding at the sensor end (pc at >>>>> the other so no decoding limitation there), thinking Send as packed >>>>> BCD of the tenths and hundredths and then hardware encode onto MSB >>>>> only when it is 1001 "9", the units 0to 3 reading as 2bit binary on >>>>> the D6 and D7 bits. Then on the pc "stick or twist" the units logged >>>>> according to whether the next tenth sent reading is 8 or 0. >>>>> I suppose in the general case 9.99 could be encoded like this for 0 to >>>>> 9.99 using D1 and D2 on the LSB but would not be reliable for general >>>>> data or jittering around the .9. >>>>> Any other harware conversion/encoding idea I might consider? >>>> >>>> I can't follow what you are doing. Are you trying to send all 400 >>>> possible combinations of 0.00 to 3.99 in just 8 bits? >>>> >>>> I have absolutely no idea what you mean by "stick or twist". >>>> >>>> Are you just trying to compress your data? >>>> >>> Don't know if it is compressing or encoding but using some of the easily >>> resolvable "illegal" ABCDEF codes of the otherwise unused bits in BCD to >>> encode the units data and then use the pc to extract and manipulate back >>> to 0 to 3.99. >> >> Your title says "2.5 digit BCD to 8-bit encoding". >> >> You can only encode 256 different values in 8 bits. >> >> There are 400 values in 0.00 to 3.99. >> >> Do you see a conflict there? >> >> I _think_ you're talking about encoding 2.5 digit BCD into at least two, >> or maybe three bytes -- but you don't SAY this. > > Of course 2-1/2 digits is only 0 to 199, which will fit fine. 2-3/4 > digits (0-399) not so much.
It is curious marketeering calling it 2-1/2 or 2-3/4 digits when they are in reality a shade under 2-1/3 and 2-2/3 respectively. Lg10(2) ~= 0.301 < 1/3 << 1/2 If the OP sends his 0 to 199 data then he can reserve the other 56 codes for sending up to 5 independent bits of status information back as well at the expense of sending a byte with no actual reading in it. Limiting readings to 0 to 191 range would allow 6 bits of status codes. -- Regards, Martin Brown
On 19/11/2014 07:10, rickman wrote:
> On 11/18/2014 6:06 PM, Tim Wescott wrote: >> On Tue, 18 Nov 2014 13:47:21 -0800, David Platt wrote: >> >>> In article <StydneML0JXVAvbJnZ2dnUU7-KednZ2d@giganews.com>, >>> Tim Wescott <seemywebsite@myfooter.really> wrote: >>> >>>> Somewhere in the hash of his original posting he mentions that the data >>>> is changing slowly. >>>> >>>> If you used one bit for a flag, you could send 128 different values of >>>> absolute data to initialize, then send a delta in the range -64 +63 (if >>>> you used two's compliment) at every sample that doesn't happen to fall >>>> on a legal "absolute" value. >>>> >>>> If your "absolute" data ranged from 0.00 to 5.08 in steps of 0.04, then >>>> as long as the data really did move slowly, you'd never be off by more >>>> than 0.02, and then only for one transmission time. >>> >>> That would be akin to an ADPCM (adaptive delta PCM) encoding, and yes, >>> it could work reasonably well. If you're strictly limited to sending >>> only one byte per sample time, it would be "lossy compression" due to >>> the slight inaccuracies it would develop from time to time, but it might >>> well be "good enough for folk music". >> >> I think it'd be much smarter to just send the reading, in ASCII, with a >> linefeed at the end to boot -- that way you plug the module into a serial >> port and look at the result on a terminal when you're debugging problems. >> >> But the OP's prose is somewhat impenetrable. > > Actually I think I cracked the code. He is thinking that he can utilize > some of the "don't care" bits that occur in the two digits, '8' and '9'. > Not fully realizing that these can indeed be used... if these digits > are transmitted. But when the two lsds are characters other than an 8 > or 9 there are no don't care bits so you lose that capacity. >
yes, perhaps I should have termed them don't-care rather than illegal. In hardware terms , for the send end, AND D4,D7 (active H for tenths digit 9) and output goes to feed the control of 2 analogue gates that pass the 2 bits of the 0,1,2,or3 units data onto lines D5,D6. Convert to serial, and pass to pc where look-up or whatever will decript back to 9 and the units figure