Forums

Help Picking Right Programming Tools

Started by ChesterW April 18, 2015
I need to pick a programming tool to plot some measurement data in real 
time. The data is transmitted from an instrument to a PC via Ethernet. 
Every second a new file is created on the PC HDD to store 1 second of 
data. The files are labeled data0, data1, etc..

Here's an example of some data:

https://dl.dropboxusercontent.com/u/33776033/exdat/data0.txt

and here's a quick look at a few bytes of data in hex:

00 AA AA AA AA AA AA 00 00 1F 1C 00 1F D4 00 1F 78 00 1F 7C 00 1F D2 00 
1F 24 00 1F A0 00 20 12 00 20 34 00 20 2E 00 1F C6 00 20 7C 00 1F CA 00 
1F F4 00 1F BE 00 20 74 00 9B B4 00 52 78 00 62 64 00 42 68 00 A3 CC 00 
F3 CC 00 42 6E 00 90 98 01 10 72 00 5E 8E 00 83 A8 00 4F 7C 00 6A 90 00 
51 48 00 7A F4 08 AD 68 00 FF FF FF FF FF FF 00 00 1E CC 00 1E AC 00 1F 
60 00 1F 2A 00 1E FA 00 1F 12 00 1F 72 00 1F 54 00 1F 58 00 1F 3C 00 1E 
C6 00 1E CC 00 1E F6 00 1F 0C 00 1F 0E 00 1E F6 00 74 00 00 2D D4 00 19 
88 00 73 68 00 93 86 01 1A 66 00 49 5E 00 3E B0 01 97 60 00 53 30 00 9D 
D2 00 20 78 00 2B 54 00 3F 0E 00 BD DA 0E 05 5A 00 FF FF FF FF FF FF 00 
00 1D B8 00 1E 3E 00 1E 38 00 1E 62 00 1D E2 00 1D EE 00 1E 86 00 1E A4 
00 1E D8 00 1E DA 00 1E 78 00 1E 56 00 1E 9E 00 1E AE 00 1E 72 00 1F 58 
00 61 90 00 15 38 00 00 00 00 66 7C 00 7F CC 00 C8 74 00 74 9E 00 29 F8 
01 9C D6 00 4F 86 00 9F 9C 00 00 00 00 1E 70 00 3D B2 01 2A AC 0E 09 64 
00 FF FF FF FF FF FF 00


The data is organized in blocks. Each block contains 32 measurements of 
20 bits each, with each 20 bit value allocated 3 bytes. The start of 
each block is delimited by hex 00 FF FF FF FF FF FF 00. 64 such blocks 
make up a complete data set. The beginning of a complete data set is 
delimited by hex 00 AA AA AA AA AA AA 00. Each second of data (i.e. each 
file) contains 5 complete data sets.

A single plotted data point consists of a linear combination (additions 
and subtractions only) of a subset of the 20 bit values in a complete 
data set. Each of these subsets contains 128 20 bit values, and there 
are 32 possible data points, i.e. there can be up to 32 traces to plot, 
so that is up to 4096 additions/subtractions per complete data set. 
Since there are 5 of these per second, that's about 20.5k adds/subtracts 
per second to produce the 160 plotted data points per second.

Ideally, I would like to plot up to 32 traces on a scrolling chart(s) in 
real time (i.e. at 5 points per second per trace).

The states of the program might look something like this:

0. configure which traces to plot and length of time axis
1. wait until data0 file is available
2. calculate linear combinations and extend plot traces
3. autoscale y axis
4. wait until next data file is available and loop to step 1.

I'll probably want to add a simple digital filter later.

So, does anyone have a suggestion about what would be a good tool to use 
to make the plots program? I'm not an experienced PC programmer, but 
would be willing to invest some time learning tools if I thought they 
would be useful in the future. I'll need to distribute the program to 
off-site users, so making a .exe is preferred, but not an absolute 
requirement.

-- 

Best Regards,

ChesterW
+++
Dr Chester Wildey
Founder MRRA Inc.
Electronic and Optoelectronic Instruments
MRI Motion, fNIRS Brain Scanners, Counterfeit and Covert Marker Detection
Fort Worth, Texas, USA
www.mrrainc.com
wildey at mrrainc dot com
ChesterW <iamsnoozin@yahoo.com> writes:

> I need to pick a programming tool to plot some measurement data in > real time. The data is transmitted from an instrument to a PC via > Ethernet. Every second a new file is created on the PC HDD to store 1 > second of data. The files are labeled data0, data1, etc.. > > Here's an example of some data: > > https://dl.dropboxusercontent.com/u/33776033/exdat/data0.txt > > and here's a quick look at a few bytes of data in hex: > > 00 AA AA AA AA AA AA 00 00 1F 1C 00 1F D4 00 1F 78 00 1F 7C 00 1F D2 > 00 1F 24 00 1F A0 00 20 12 00 20 34 00 20 2E 00 1F C6 00 20 7C 00 1F > CA 00 1F F4 00 1F BE 00 20 74 00 9B B4 00 52 78 00 62 64 00 42 68 00 > A3 CC 00 F3 CC 00 42 6E 00 90 98 01 10 72 00 5E 8E 00 83 A8 00 4F 7C > 00 6A 90 00 51 48 00 7A F4 08 AD 68 00 FF FF FF FF FF FF 00 00 1E CC > 00 1E AC 00 1F 60 00 1F 2A 00 1E FA 00 1F 12 00 1F 72 00 1F 54 00 1F > 58 00 1F 3C 00 1E C6 00 1E CC 00 1E F6 00 1F 0C 00 1F 0E 00 1E F6 00 > 74 00 00 2D D4 00 19 88 00 73 68 00 93 86 01 1A 66 00 49 5E 00 3E B0 > 01 97 60 00 53 30 00 9D D2 00 20 78 00 2B 54 00 3F 0E 00 BD DA 0E 05 > 5A 00 FF FF FF FF FF FF 00 00 1D B8 00 1E 3E 00 1E 38 00 1E 62 00 1D > E2 00 1D EE 00 1E 86 00 1E A4 00 1E D8 00 1E DA 00 1E 78 00 1E 56 00 > 1E 9E 00 1E AE 00 1E 72 00 1F 58 00 61 90 00 15 38 00 00 00 00 66 7C > 00 7F CC 00 C8 74 00 74 9E 00 29 F8 01 9C D6 00 4F 86 00 9F 9C 00 00 > 00 00 1E 70 00 3D B2 01 2A AC 0E 09 64 00 FF FF FF FF FF FF 00 > > > The data is organized in blocks. Each block contains 32 measurements > of 20 bits each, with each 20 bit value allocated 3 bytes. The start > of each block is delimited by hex 00 FF FF FF FF FF FF 00. 64 such > blocks make up a complete data set. The beginning of a complete data > set is delimited by hex 00 AA AA AA AA AA AA 00. Each second of data > (i.e. each file) contains 5 complete data sets. > > A single plotted data point consists of a linear combination > (additions and subtractions only) of a subset of the 20 bit values in > a complete data set. Each of these subsets contains 128 20 bit values, > and there are 32 possible data points, i.e. there can be up to 32 > traces to plot, so that is up to 4096 additions/subtractions per > complete data set. Since there are 5 of these per second, that's about > 20.5k adds/subtracts per second to produce the 160 plotted data points > per second.
No problem in any language these days
> > Ideally, I would like to plot up to 32 traces on a scrolling chart(s) > in real time (i.e. at 5 points per second per trace). > > The states of the program might look something like this: > > 0. configure which traces to plot and length of time axis > 1. wait until data0 file is available > 2. calculate linear combinations and extend plot traces > 3. autoscale y axis > 4. wait until next data file is available and loop to step 1. > > I'll probably want to add a simple digital filter later. > > So, does anyone have a suggestion about what would be a good tool to > use to make the plots program? I'm not an experienced PC programmer, > but would be willing to invest some time learning tools if I thought > they would be useful in the future. I'll need to distribute the > program to off-site users, so making a .exe is preferred, but not an > absolute requirement.
Perhaps Python, it is fairly easy to learn and there are a lot of scientific graphics libraries for it. It is free and cross-platform too. Regarding the single .exe there are probably packaging solutions to do this for you or you can distribute python separately. Everyone has their favorite tool for this sort of thing, I am sure you could use matlab, C#, java, whatever. -- John Devereux
On 4/18/2015 12:00 AM, ChesterW wrote:
> I need to pick a programming tool to plot some measurement data in real time. > The data is transmitted from an instrument to a PC via Ethernet. Every second a > new file is created on the PC HDD to store 1 second of data. The files are > labeled data0, data1, etc..
What crates these files? Is it something you have control over (i.e. could it take on this added responsibility)?
> Here's an example of some data: > > https://dl.dropboxusercontent.com/u/33776033/exdat/data0.txt > > and here's a quick look at a few bytes of data in hex: > > 00 AA AA AA AA AA AA 00 00 1F 1C 00 1F D4 00 1F 78 00 1F 7C 00 1F D2 00 1F 24 > 00 1F A0 00 20 12 00 20 34 00 20 2E 00 1F C6 00 20 7C 00 1F CA 00 1F F4 00 1F > BE 00 20 74 00 9B B4 00 52 78 00 62 64 00 42 68 00 A3 CC 00 F3 CC 00 42 6E 00 > 90 98 01 10 72 00 5E 8E 00 83 A8 00 4F 7C 00 6A 90 00 51 48 00 7A F4 08 AD 68 > 00 FF FF FF FF FF FF 00 00 1E CC 00 1E AC 00 1F 60 00 1F 2A 00 1E FA 00 1F 12 > 00 1F 72 00 1F 54 00 1F 58 00 1F 3C 00 1E C6 00 1E CC 00 1E F6 00 1F 0C 00 1F > 0E 00 1E F6 00 74 00 00 2D D4 00 19 88 00 73 68 00 93 86 01 1A 66 00 49 5E 00 > 3E B0 01 97 60 00 53 30 00 9D D2 00 20 78 00 2B 54 00 3F 0E 00 BD DA 0E 05 5A > 00 FF FF FF FF FF FF 00 00 1D B8 00 1E 3E 00 1E 38 00 1E 62 00 1D E2 00 1D EE > 00 1E 86 00 1E A4 00 1E D8 00 1E DA 00 1E 78 00 1E 56 00 1E 9E 00 1E AE 00 1E > 72 00 1F 58 00 61 90 00 15 38 00 00 00 00 66 7C 00 7F CC 00 C8 74 00 74 9E 00 > 29 F8 01 9C D6 00 4F 86 00 9F 9C 00 00 00 00 1E 70 00 3D B2 01 2A AC 0E 09 64 > 00 FF FF FF FF FF FF 00 > > The data is organized in blocks. Each block contains 32 measurements of 20 bits > each, with each 20 bit value allocated 3 bytes. The start of each block is > delimited by hex 00 FF FF FF FF FF FF 00. 64 such blocks make up a complete > data set. The beginning of a complete data set is delimited by hex 00 AA AA AA > AA AA AA 00. Each second of data (i.e. each file) contains 5 complete data sets.
So, a block is 96 bytes (32*3) of data and an 8 byte header? But, the *first* block is NOT prefaced by the "00 FF FF FF FF FF FF 00" BLOCK_HEADER? Instead, the "00 AA AA AA AA AA AA 00" DATASET_HEADER acts to introduce that block? Or, are the 96 blocks of "garbage" in your above example -- after the DATASET_HEADER but before the first BLOCK_HEADER? (i.e., is the BLOCK_HEADER required before each block? Or, does the DATASET_HEADER also function as a BLOCK_HEADER?) A dataset contains 2048 (64*32) "measurements" encoded in 6144 bytes (accompanied by 512 bytes of headers).
> A single plotted data point consists of a linear combination (additions and > subtractions only) of a subset of the 20 bit values in a complete data set.
Do you mean "3 byte measurement in block B1 at offset X1" plus/minus the "3 byte measurement in block B2 at offset X1" plus/minus... ?
> Each of these subsets contains 128 20 bit values, and there are 32 possible > data points, i.e. there can be up to 32 traces to plot, so that is up to 4096 > additions/subtractions per complete data set. Since there are 5 of these per > second, that's about 20.5k adds/subtracts per second to produce the 160 plotted > data points per second. > > Ideally, I would like to plot up to 32 traces on a scrolling chart(s) in real > time (i.e. at 5 points per second per trace). > > The states of the program might look something like this: > > 0. configure which traces to plot and length of time axis
I.e., figure out which Bi and Xi to combine into each "plotted value"?
> 1. wait until data0 file is available > 2. calculate linear combinations and extend plot traces > 3. autoscale y axis > 4. wait until next data file is available and loop to step 1. > > I'll probably want to add a simple digital filter later.
What are the consequences of failing to "keep up" with the data? Or, of not producing "smooth"/continuous motion in the "strip chart"? (i.e., what if data5 is sluggish to be plotted and appears 1.8 seconds after data4 has been plotted; but, data6 is plotted right on time -- at 2.0 seconds after data4 was plotted. So, data5 is only visible for 0.2 seconds) What happens if data is *dropped* (i.e., a discontinuity in the plot)?
> So, does anyone have a suggestion about what would be a good tool to use to > make the plots program? I'm not an experienced PC programmer, but would be > willing to invest some time learning tools if I thought they would be useful in > the future. I'll need to distribute the program to off-site users, so making a > .exe is preferred, but not an absolute requirement.
How important is the "live" aspect of the presentation? Could you, instead, accept an "offline" implementation: gather up all the dataX files and then process them at some later time -- giving the user the ability to scroll through the LONG chart?
On Saturday, April 18, 2015 at 3:00:16 AM UTC-4, ChesterW wrote:
> I need to pick a programming tool to plot some measurement data in real > time. The data is transmitted from an instrument to a PC via Ethernet. > Every second a new file is created on the PC HDD to store 1 second of > data. The files are labeled data0, data1, etc.. > > Here's an example of some data: > > https://dl.dropboxusercontent.com/u/33776033/exdat/data0.txt > > and here's a quick look at a few bytes of data in hex: > > 00 AA AA AA AA AA AA 00 00 1F 1C 00 1F D4 00 1F 78 00 1F 7C 00 1F D2 00 > 1F 24 00 1F A0 00 20 12 00 20 34 00 20 2E 00 1F C6 00 20 7C 00 1F CA 00 > 1F F4 00 1F BE 00 20 74 00 9B B4 00 52 78 00 62 64 00 42 68 00 A3 CC 00 > F3 CC 00 42 6E 00 90 98 01 10 72 00 5E 8E 00 83 A8 00 4F 7C 00 6A 90 00 > 51 48 00 7A F4 08 AD 68 00 FF FF FF FF FF FF 00 00 1E CC 00 1E AC 00 1F > 60 00 1F 2A 00 1E FA 00 1F 12 00 1F 72 00 1F 54 00 1F 58 00 1F 3C 00 1E > C6 00 1E CC 00 1E F6 00 1F 0C 00 1F 0E 00 1E F6 00 74 00 00 2D D4 00 19 > 88 00 73 68 00 93 86 01 1A 66 00 49 5E 00 3E B0 01 97 60 00 53 30 00 9D > D2 00 20 78 00 2B 54 00 3F 0E 00 BD DA 0E 05 5A 00 FF FF FF FF FF FF 00 > 00 1D B8 00 1E 3E 00 1E 38 00 1E 62 00 1D E2 00 1D EE 00 1E 86 00 1E A4 > 00 1E D8 00 1E DA 00 1E 78 00 1E 56 00 1E 9E 00 1E AE 00 1E 72 00 1F 58 > 00 61 90 00 15 38 00 00 00 00 66 7C 00 7F CC 00 C8 74 00 74 9E 00 29 F8 > 01 9C D6 00 4F 86 00 9F 9C 00 00 00 00 1E 70 00 3D B2 01 2A AC 0E 09 64 > 00 FF FF FF FF FF FF 00 > > > The data is organized in blocks. Each block contains 32 measurements of > 20 bits each, with each 20 bit value allocated 3 bytes. The start of > each block is delimited by hex 00 FF FF FF FF FF FF 00. 64 such blocks > make up a complete data set. The beginning of a complete data set is > delimited by hex 00 AA AA AA AA AA AA 00. Each second of data (i.e. each > file) contains 5 complete data sets. > > A single plotted data point consists of a linear combination (additions > and subtractions only) of a subset of the 20 bit values in a complete > data set. Each of these subsets contains 128 20 bit values, and there > are 32 possible data points, i.e. there can be up to 32 traces to plot, > so that is up to 4096 additions/subtractions per complete data set. > Since there are 5 of these per second, that's about 20.5k adds/subtracts > per second to produce the 160 plotted data points per second. > > Ideally, I would like to plot up to 32 traces on a scrolling chart(s) in > real time (i.e. at 5 points per second per trace). > > The states of the program might look something like this: > > 0. configure which traces to plot and length of time axis > 1. wait until data0 file is available > 2. calculate linear combinations and extend plot traces > 3. autoscale y axis > 4. wait until next data file is available and loop to step 1. > > I'll probably want to add a simple digital filter later. > > So, does anyone have a suggestion about what would be a good tool to use > to make the plots program? I'm not an experienced PC programmer, but > would be willing to invest some time learning tools if I thought they > would be useful in the future. I'll need to distribute the program to > off-site users, so making a .exe is preferred, but not an absolute > requirement. > > -- > > Best Regards, > > ChesterW > +++ > Dr Chester Wildey > Founder MRRA Inc. > Electronic and Optoelectronic Instruments > MRI Motion, fNIRS Brain Scanners, Counterfeit and Covert Marker Detection > Fort Worth, Texas, USA > www.mrrainc.com > wildey at mrrainc dot com
You could do it in Python. Here's a script to get you started. It converts your hex file into csv file can pull into Excel. # hex2csv import os def hex2csv(filename): """ Convert an binary hex file to an ascii csv file filename -- The path to the binary hex file generates an ascii csv file """ basename = os.path.splitext(filename)[0] outfilename = basename +'.csv' outfile = open(outfilename, 'wb') infile = open(filename, 'rb') for line in infile: for ch in line: outfile.write( str(int(ord(ch))) + ',') infile.close() outfile.close() def main(): """ Method called from command line """ print 'Convert a binary hex file to an ascii csv file' filename = raw_input('Enter Path to File: ') print filename hex2csv(filename) if __name__ == '__main__': main() Your next steps would be to write programs to break the files into your blocks, data and whatever. Process the data with numpy. Then you can plot it with matplotlib. Then if you need a GUI you can use wxPython or one of the other flavors. I would check the answers in each step against what you get processing the data in Excel or one of the freeware spreadsheets. Have fun.
ChesterW wrote:
> I need to pick a programming tool to plot some measurement data in real > time. The data is transmitted from an instrument to a PC via Ethernet. > Every second a new file is created on the PC HDD to store 1 second of > data. The files are labeled data0, data1, etc.. > > Here's an example of some data: > > https://dl.dropboxusercontent.com/u/33776033/exdat/data0.txt > > and here's a quick look at a few bytes of data in hex: > > 00 AA AA AA AA AA AA 00 00 1F 1C 00 1F D4 00 1F 78 00 1F 7C 00 1F D2 00 > 1F 24 00 1F A0 00 20 12 00 20 34 00 20 2E 00 1F C6 00 20 7C 00 1F CA 00 > 1F F4 00 1F BE 00 20 74 00 9B B4 00 52 78 00 62 64 00 42 68 00 A3 CC 00 > F3 CC 00 42 6E 00 90 98 01 10 72 00 5E 8E 00 83 A8 00 4F 7C 00 6A 90 00 > 51 48 00 7A F4 08 AD 68 00 FF FF FF FF FF FF 00 00 1E CC 00 1E AC 00 1F > 60 00 1F 2A 00 1E FA 00 1F 12 00 1F 72 00 1F 54 00 1F 58 00 1F 3C 00 1E > C6 00 1E CC 00 1E F6 00 1F 0C 00 1F 0E 00 1E F6 00 74 00 00 2D D4 00 19 > 88 00 73 68 00 93 86 01 1A 66 00 49 5E 00 3E B0 01 97 60 00 53 30 00 9D > D2 00 20 78 00 2B 54 00 3F 0E 00 BD DA 0E 05 5A 00 FF FF FF FF FF FF 00 > 00 1D B8 00 1E 3E 00 1E 38 00 1E 62 00 1D E2 00 1D EE 00 1E 86 00 1E A4 > 00 1E D8 00 1E DA 00 1E 78 00 1E 56 00 1E 9E 00 1E AE 00 1E 72 00 1F 58 > 00 61 90 00 15 38 00 00 00 00 66 7C 00 7F CC 00 C8 74 00 74 9E 00 29 F8 > 01 9C D6 00 4F 86 00 9F 9C 00 00 00 00 1E 70 00 3D B2 01 2A AC 0E 09 64 > 00 FF FF FF FF FF FF 00 > > > The data is organized in blocks. Each block contains 32 measurements of > 20 bits each, with each 20 bit value allocated 3 bytes. The start of > each block is delimited by hex 00 FF FF FF FF FF FF 00. 64 such blocks > make up a complete data set. The beginning of a complete data set is > delimited by hex 00 AA AA AA AA AA AA 00. Each second of data (i.e. each > file) contains 5 complete data sets. > > A single plotted data point consists of a linear combination (additions > and subtractions only) of a subset of the 20 bit values in a complete > data set. Each of these subsets contains 128 20 bit values, and there > are 32 possible data points, i.e. there can be up to 32 traces to plot, > so that is up to 4096 additions/subtractions per complete data set. > Since there are 5 of these per second, that's about 20.5k adds/subtracts > per second to produce the 160 plotted data points per second. > > Ideally, I would like to plot up to 32 traces on a scrolling chart(s) in > real time (i.e. at 5 points per second per trace). > > The states of the program might look something like this: > > 0. configure which traces to plot and length of time axis > 1. wait until data0 file is available > 2. calculate linear combinations and extend plot traces > 3. autoscale y axis > 4. wait until next data file is available and loop to step 1. > > I'll probably want to add a simple digital filter later. > > So, does anyone have a suggestion about what would be a good tool to use > to make the plots program? I'm not an experienced PC programmer, but > would be willing to invest some time learning tools if I thought they > would be useful in the future. I'll need to distribute the program to > off-site users, so making a .exe is preferred, but not an absolute > requirement. >
Data rate is slow enough to allow interpreted BASIC.
ChesterW <iamsnoozin@yahoo.com> wrote:
> I need to pick a programming tool to plot some measurement data in > real time.
The first thing I thought of: Perl script to watch for new data files, read them, and parse them. Gnuplot (controlled by the Perl script) to plot the data into a graphics file (PNG or similar). Image viewing program that will display the graphics file and refresh the display whenever the file changes. I am not quite sure which image viewing program to use. IrfanView has many options and may meet the requirements. If you don't mind limiting GUI updates to 1 Hz, you could use a Web browser as the image display program, with a little bit of static HTML to make it refresh the image. This would give you a graph that updates, but users would not easily be able to pause the updates (while parsing and further plotting continues in the background), or scroll the graph around to examine previous data. The Perl script can optionally write the parsed data to a CSV or Excel file, if that might be useful later.
> I'll need to distribute the program to off-site users, so making a > .exe is preferred, but not an absolute requirement.
For many of the proposed solutions, including mine, this is kind of a tough one. You'll need an interpreter (Perl for me, Python for many others), Gnuplot, and some kind of image viewing program. It is possible to package the installers for all of these into a single .exe. This requires your users to have privileges to install programs, which may not always be available. It also turns the process into "run installer, locate icon for installed script, click on it" rather than "run the provided .exe file and start plotting data immediately". The leading alternative would be to write a custom application, probably in C++ or C#. There is probably a chart widget that would do the plotting for you; you'd get to roll your own file parsing code. Matt Roberds
Den l&#2013266168;rdag den 18. april 2015 kl. 09.00.16 UTC+2 skrev ChesterW:
> I need to pick a programming tool to plot some measurement data in real > time. The data is transmitted from an instrument to a PC via Ethernet. > Every second a new file is created on the PC HDD to store 1 second of > data. The files are labeled data0, data1, etc.. > > Here's an example of some data: > > https://dl.dropboxusercontent.com/u/33776033/exdat/data0.txt > > and here's a quick look at a few bytes of data in hex: > > 00 AA AA AA AA AA AA 00 00 1F 1C 00 1F D4 00 1F 78 00 1F 7C 00 1F D2 00 > 1F 24 00 1F A0 00 20 12 00 20 34 00 20 2E 00 1F C6 00 20 7C 00 1F CA 00 > 1F F4 00 1F BE 00 20 74 00 9B B4 00 52 78 00 62 64 00 42 68 00 A3 CC 00 > F3 CC 00 42 6E 00 90 98 01 10 72 00 5E 8E 00 83 A8 00 4F 7C 00 6A 90 00 > 51 48 00 7A F4 08 AD 68 00 FF FF FF FF FF FF 00 00 1E CC 00 1E AC 00 1F > 60 00 1F 2A 00 1E FA 00 1F 12 00 1F 72 00 1F 54 00 1F 58 00 1F 3C 00 1E > C6 00 1E CC 00 1E F6 00 1F 0C 00 1F 0E 00 1E F6 00 74 00 00 2D D4 00 19 > 88 00 73 68 00 93 86 01 1A 66 00 49 5E 00 3E B0 01 97 60 00 53 30 00 9D > D2 00 20 78 00 2B 54 00 3F 0E 00 BD DA 0E 05 5A 00 FF FF FF FF FF FF 00 > 00 1D B8 00 1E 3E 00 1E 38 00 1E 62 00 1D E2 00 1D EE 00 1E 86 00 1E A4 > 00 1E D8 00 1E DA 00 1E 78 00 1E 56 00 1E 9E 00 1E AE 00 1E 72 00 1F 58 > 00 61 90 00 15 38 00 00 00 00 66 7C 00 7F CC 00 C8 74 00 74 9E 00 29 F8 > 01 9C D6 00 4F 86 00 9F 9C 00 00 00 00 1E 70 00 3D B2 01 2A AC 0E 09 64 > 00 FF FF FF FF FF FF 00 > > > The data is organized in blocks. Each block contains 32 measurements of > 20 bits each, with each 20 bit value allocated 3 bytes. The start of > each block is delimited by hex 00 FF FF FF FF FF FF 00. 64 such blocks > make up a complete data set. The beginning of a complete data set is > delimited by hex 00 AA AA AA AA AA AA 00. Each second of data (i.e. each > file) contains 5 complete data sets. > > A single plotted data point consists of a linear combination (additions > and subtractions only) of a subset of the 20 bit values in a complete > data set. Each of these subsets contains 128 20 bit values, and there > are 32 possible data points, i.e. there can be up to 32 traces to plot, > so that is up to 4096 additions/subtractions per complete data set. > Since there are 5 of these per second, that's about 20.5k adds/subtracts > per second to produce the 160 plotted data points per second. > > Ideally, I would like to plot up to 32 traces on a scrolling chart(s) in > real time (i.e. at 5 points per second per trace). > > The states of the program might look something like this: > > 0. configure which traces to plot and length of time axis > 1. wait until data0 file is available > 2. calculate linear combinations and extend plot traces > 3. autoscale y axis > 4. wait until next data file is available and loop to step 1. > > I'll probably want to add a simple digital filter later. > > So, does anyone have a suggestion about what would be a good tool to use > to make the plots program? I'm not an experienced PC programmer, but > would be willing to invest some time learning tools if I thought they > would be useful in the future. I'll need to distribute the program to > off-site users, so making a .exe is preferred, but not an absolute > requirement. >
would be straight forward with something like octave -Lasse
On 4/18/15, 3:49 AM, John Devereux wrote:
> ChesterW <iamsnoozin@yahoo.com> writes: > >> I need to pick a programming tool to plot some measurement data in >> real time. The data is transmitted from an instrument to a PC via >> Ethernet. Every second a new file is created on the PC HDD to store 1 >> second of data. The files are labeled data0, data1, etc.. >> >> Here's an example of some data: >> >> https://dl.dropboxusercontent.com/u/33776033/exdat/data0.txt >> >> and here's a quick look at a few bytes of data in hex: >> >> 00 AA AA AA AA AA AA 00 00 1F 1C 00 1F D4 00 1F 78 00 1F 7C 00 1F D2 >> 00 1F 24 00 1F A0 00 20 12 00 20 34 00 20 2E 00 1F C6 00 20 7C 00 1F >> CA 00 1F F4 00 1F BE 00 20 74 00 9B B4 00 52 78 00 62 64 00 42 68 00 >> A3 CC 00 F3 CC 00 42 6E 00 90 98 01 10 72 00 5E 8E 00 83 A8 00 4F 7C >> 00 6A 90 00 51 48 00 7A F4 08 AD 68 00 FF FF FF FF FF FF 00 00 1E CC >> 00 1E AC 00 1F 60 00 1F 2A 00 1E FA 00 1F 12 00 1F 72 00 1F 54 00 1F >> 58 00 1F 3C 00 1E C6 00 1E CC 00 1E F6 00 1F 0C 00 1F 0E 00 1E F6 00 >> 74 00 00 2D D4 00 19 88 00 73 68 00 93 86 01 1A 66 00 49 5E 00 3E B0 >> 01 97 60 00 53 30 00 9D D2 00 20 78 00 2B 54 00 3F 0E 00 BD DA 0E 05 >> 5A 00 FF FF FF FF FF FF 00 00 1D B8 00 1E 3E 00 1E 38 00 1E 62 00 1D >> E2 00 1D EE 00 1E 86 00 1E A4 00 1E D8 00 1E DA 00 1E 78 00 1E 56 00 >> 1E 9E 00 1E AE 00 1E 72 00 1F 58 00 61 90 00 15 38 00 00 00 00 66 7C >> 00 7F CC 00 C8 74 00 74 9E 00 29 F8 01 9C D6 00 4F 86 00 9F 9C 00 00 >> 00 00 1E 70 00 3D B2 01 2A AC 0E 09 64 00 FF FF FF FF FF FF 00 >> >> >> The data is organized in blocks. Each block contains 32 measurements >> of 20 bits each, with each 20 bit value allocated 3 bytes. The start >> of each block is delimited by hex 00 FF FF FF FF FF FF 00. 64 such >> blocks make up a complete data set. The beginning of a complete data >> set is delimited by hex 00 AA AA AA AA AA AA 00. Each second of data >> (i.e. each file) contains 5 complete data sets. >> >> A single plotted data point consists of a linear combination >> (additions and subtractions only) of a subset of the 20 bit values in >> a complete data set. Each of these subsets contains 128 20 bit values, >> and there are 32 possible data points, i.e. there can be up to 32 >> traces to plot, so that is up to 4096 additions/subtractions per >> complete data set. Since there are 5 of these per second, that's about >> 20.5k adds/subtracts per second to produce the 160 plotted data points >> per second. > > No problem in any language these days > >> >> Ideally, I would like to plot up to 32 traces on a scrolling chart(s) >> in real time (i.e. at 5 points per second per trace). >> >> The states of the program might look something like this: >> >> 0. configure which traces to plot and length of time axis >> 1. wait until data0 file is available >> 2. calculate linear combinations and extend plot traces >> 3. autoscale y axis >> 4. wait until next data file is available and loop to step 1. >> >> I'll probably want to add a simple digital filter later. >> >> So, does anyone have a suggestion about what would be a good tool to >> use to make the plots program? I'm not an experienced PC programmer, >> but would be willing to invest some time learning tools if I thought >> they would be useful in the future. I'll need to distribute the >> program to off-site users, so making a .exe is preferred, but not an >> absolute requirement. > > Perhaps Python, it is fairly easy to learn and there are a lot of > scientific graphics libraries for it. It is free and cross-platform too. > > Regarding the single .exe there are probably packaging solutions to do > this for you or you can distribute python separately. > > Everyone has their favorite tool for this sort of thing, I am sure you > could use matlab, C#, java, whatever. > >
Thanks John. I'll give Python a look. -- Best Regards, ChesterW
Hi Don,

On 4/18/15, 4:15 AM, Don Y wrote:
> On 4/18/2015 12:00 AM, ChesterW wrote:
<snip>
> > What crates these files? Is it something you have control over (i.e. > could it > take on this added responsibility)?
The data comes over Ethernet from an embedded system. A Java program on the PC parses and saves the files. I hired a guy to do the plotting program in Java. He came up with a graphics package and wrote the program, but it couldn't keep up.
> >> Here's an example of some data: >> >> https://dl.dropboxusercontent.com/u/33776033/exdat/data0.txt >> >> and here's a quick look at a few bytes of data in hex: >> >> 00 AA AA AA AA AA AA 00 00 1F 1C 00 1F D4 00 1F 78 00 1F 7C 00 1F D2 >> 00 1F 24 >> 00 1F A0 00 20 12 00 20 34 00 20 2E 00 1F C6 00 20 7C 00 1F CA 00 1F >> F4 00 1F >> BE 00 20 74 00 9B B4 00 52 78 00 62 64 00 42 68 00 A3 CC 00 F3 CC 00 >> 42 6E 00 >> 90 98 01 10 72 00 5E 8E 00 83 A8 00 4F 7C 00 6A 90 00 51 48 00 7A F4 >> 08 AD 68 >> 00 FF FF FF FF FF FF 00 00 1E CC 00 1E AC 00 1F 60 00 1F 2A 00 1E FA >> 00 1F 12 >> 00 1F 72 00 1F 54 00 1F 58 00 1F 3C 00 1E C6 00 1E CC 00 1E F6 00 1F >> 0C 00 1F >> 0E 00 1E F6 00 74 00 00 2D D4 00 19 88 00 73 68 00 93 86 01 1A 66 00 >> 49 5E 00 >> 3E B0 01 97 60 00 53 30 00 9D D2 00 20 78 00 2B 54 00 3F 0E 00 BD DA >> 0E 05 5A >> 00 FF FF FF FF FF FF 00 00 1D B8 00 1E 3E 00 1E 38 00 1E 62 00 1D E2 >> 00 1D EE >> 00 1E 86 00 1E A4 00 1E D8 00 1E DA 00 1E 78 00 1E 56 00 1E 9E 00 1E >> AE 00 1E >> 72 00 1F 58 00 61 90 00 15 38 00 00 00 00 66 7C 00 7F CC 00 C8 74 00 >> 74 9E 00 >> 29 F8 01 9C D6 00 4F 86 00 9F 9C 00 00 00 00 1E 70 00 3D B2 01 2A AC >> 0E 09 64 >> 00 FF FF FF FF FF FF 00 >> >> The data is organized in blocks. Each block contains 32 measurements >> of 20 bits >> each, with each 20 bit value allocated 3 bytes. The start of each >> block is >> delimited by hex 00 FF FF FF FF FF FF 00. 64 such blocks make up a >> complete >> data set. The beginning of a complete data set is delimited by hex 00 >> AA AA AA >> AA AA AA 00. Each second of data (i.e. each file) contains 5 complete >> data sets. > > So, a block is 96 bytes (32*3) of data and an 8 byte header?
Yes.
> > But, the *first* block is NOT prefaced by the "00 FF FF FF FF FF FF 00" > BLOCK_HEADER? Instead, the "00 AA AA AA AA AA AA 00" DATASET_HEADER acts > to introduce that block? Or, are the 96 blocks of "garbage" in your > above example -- after the DATASET_HEADER but before the first > BLOCK_HEADER? > > (i.e., is the BLOCK_HEADER required before each block? Or, does the > DATASET_HEADER also function as a BLOCK_HEADER?)
The DATASET_HEADER also functions as a BLOCK_HEADER.
> > A dataset contains 2048 (64*32) "measurements" encoded in 6144 bytes > (accompanied by 512 bytes of headers).
Yes.
> >> A single plotted data point consists of a linear combination >> (additions and >> subtractions only) of a subset of the 20 bit values in a complete data >> set. > > Do you mean "3 byte measurement in block B1 at offset X1" plus/minus > the "3 byte measurement in block B2 at offset X1" plus/minus... ?
Yes, but the choices are plus/minus/skip with ratios 0.25/0.25/0.5.
> >> Each of these subsets contains 128 20 bit values, and there are 32 >> possible >> data points, i.e. there can be up to 32 traces to plot, so that is up >> to 4096 >> additions/subtractions per complete data set. Since there are 5 of >> these per >> second, that's about 20.5k adds/subtracts per second to produce the >> 160 plotted >> data points per second. >> >> Ideally, I would like to plot up to 32 traces on a scrolling chart(s) >> in real >> time (i.e. at 5 points per second per trace). >> >> The states of the program might look something like this: >> >> 0. configure which traces to plot and length of time axis > > I.e., figure out which Bi and Xi to combine into each "plotted value"?
Yes, Bi, Xi, plus data giving the add/subtract/skip rule (a string with values of +1,-1,0). Each Bi has a unique rule string.
> >> 1. wait until data0 file is available >> 2. calculate linear combinations and extend plot traces >> 3. autoscale y axis >> 4. wait until next data file is available and loop to step 1. >> >> I'll probably want to add a simple digital filter later. > > What are the consequences of failing to "keep up" with the data?
This is a real-time display for human consumption. No terrible consequences for failure other than looking cheesy, as long as it doesn't crash the PC, that is. It is possible to use the system 'blind' without display, but it is a pain. A user could be taking blank data for 10 minutes or so before discovering the problem.
> > Or, of not producing "smooth"/continuous motion in the "strip chart"? > (i.e., what if data5 is sluggish to be plotted and appears 1.8 seconds > after data4 has been plotted; but, data6 is plotted right on time -- at > 2.0 seconds after data4 was plotted. So, data5 is only visible for 0.2
Some delay is prob OK. 0.2 seconds would prob be OK.
> seconds) > > What happens if data is *dropped* (i.e., a discontinuity in the plot)?
As long as the system takes back up when the data resumes, it would be acceptable. This assumes that no more than about a second or two of data is lost. Any more would need to signal system failure.
> >> So, does anyone have a suggestion about what would be a good tool to >> use to >> make the plots program? I'm not an experienced PC programmer, but >> would be >> willing to invest some time learning tools if I thought they would be >> useful in >> the future. I'll need to distribute the program to off-site users, so >> making a >> .exe is preferred, but not an absolute requirement. > > How important is the "live" aspect of the presentation? Could you, > instead, accept an "offline" implementation: gather up all the > dataX files and then process them at some later time -- giving > the user the ability to scroll through the LONG chart?
I currently process the data offline using an Octave script. The idea here is to make it a live display. I want to choose a tool that makes a good trade between ease-of-use and efficiency. I think I need something between Assembler and Java, I just don't know what tool. I don't mind learning something new, I'd just like to ensure that I pick a tool where success is at least possible. I'd also like to pick a tool that I could use for other tasks. I'm not a windows programmer though, so there is plenty of room for bad assumptions on my part. That's why I'm asking for your help.
>
-- Best Regards, ChesterW
On 4/18/15, 2:24 PM, Wanderer wrote:
<snip>
> You could do it in Python. Here's a script to get you started. It converts your hex file into csv file can pull into Excel. > > # hex2csv > > import os > > def hex2csv(filename): > """ Convert an binary hex file to an ascii csv file > filename -- The path to the binary hex file > generates an ascii csv file > """ > basename = os.path.splitext(filename)[0] > outfilename = basename +'.csv' > > outfile = open(outfilename, 'wb') > infile = open(filename, 'rb') > for line in infile: > for ch in line: > outfile.write( str(int(ord(ch))) + ',') > > infile.close() > outfile.close() > > def main(): > """ Method called from command line > """ > print 'Convert a binary hex file to an ascii csv file' > filename = raw_input('Enter Path to File: ') > print filename > hex2csv(filename) > > > if __name__ == '__main__': > main() > > Your next steps would be to write programs to break the files into your blocks, data and whatever. Process the data with numpy. Then you can plot it with matplotlib. Then if you need a GUI you can use wxPython or one of the other flavors. I would check the answers in each step against what you get processing the data in Excel or one of the freeware spreadsheets. > > Have fun. >
Thanks, I'll give Python a look. -- Best Regards, ChesterW