On 2019-01-21 07:51, gnuarm.deletethisbit@gmail.com wrote:> On Monday, January 21, 2019 at 10:44:34 AM UTC-5, Joerg wrote: >> On 2019-01-20 16:05, Tim Williams wrote: >>> https://softwareengineering.stackexchange.com/questions/153351/is-there-an-specific-way-or-algorithm-to-decode-protocols >>> >>> >>>?>>> >>> Tons of examples of software decode for embedded platforms, might >>> take longer to refactor for a data series than to write one new >>> though? >>> >>> I'd gladly write one say in JavaScript and put it up on my >>> website for all to use, but I don't need one, and I'm guessing >>> you don't have the time/budget to do that on request. >>> >>> Well, such is life :) >>> >> >> Yeah, time is the issue and I am now to the point where I pretty >> much have to decode by hand. > > Not really. If you send me a file I will write a program to pull out > the data. What format do you want the output? >Ideally in hex. I had sent you the file this morning. Meantime Martin and Lasse sent me code (Martin in Excel VBA with macros, Lasse in C) which seems to work. They result in the same hex output which is encouraging. Embarrassed to say but I am not familiar with compiling stuff so I'll probably try to plow ahead with Excel, or in my case try to run it in OpenOffice. -- Regards, Joerg http://www.analogconsultants.com/
RS232 decode routines from raw scope data?
Started by ●January 20, 2019
Reply by ●January 21, 20192019-01-21
Reply by ●January 21, 20192019-01-21
On 1/21/2019 7:36 AM, Joerg wrote:> On 2019-01-21 05:02, Chris Jones wrote: >> On 21/01/2019 10:06, Joerg wrote: >>> On 2019-01-20 14:49, bitrex wrote: >>>> On 01/20/2019 05:26 PM, Joerg wrote: >>>>> On 2019-01-20 13:54, Sjouke Burry wrote: >>>>>> On 20-1-2019 22:51, Sjouke Burry wrote: >>>>>>> On 20-1-2019 21:52, Joerg wrote: >>>>>>>> Is there a simple Windows or Linux program that can decode serial >>>>>>>> data >>>>>>>> out of raw data from an digital oscilloscope? >>>>>>>> >>>>>>>> My old logic analyzer does not have any serial data decoding and >>>>>>>> neither >>>>>>>> does my scope. So I'd need something that can take in the ADC raw >>>>>>>> data, >>>>>>>> I set a threshold to turn that into H/L digital values and the >>>>>>>> software >>>>>>>> would fish out the data words. Not sure if it even exist. Not >>>>>>>> something >>>>>>>> expensive because I'll only need it once for a short time, to >>>>>>>> find if >>>>>>>> there's a bug visible in the data. It'll be a challenge because my >>>>>>>> scope >>>>>>>> only has 25,000 points recording length. >>>>>>>> >>>>>>> For reading/writing the hardware ports, (for me in DOS) you >>>>>>> can download my code. >>>>>>> The exe can be loaded on both sides of a nul modem connection, >>>>>>> and gives both screen access to the oher computer. >>>>>>> >>>>>>> It uses the hardware buffer in the rs232 chip, no interrupt used. >>>>>>> >>>>>> Sorry, I forgot the link.> >>>>>> http://home.planet.nl/~burry004/COMPORT.ZIP >>>>>> >>>>> >>>>> Thanks, Sjouke, but unfortunately this isn't coming from the harware >>>>> port. It is raw data in the form of an oscilloscope data dump. >>>>> Basically 25,000 data words of 8-bits each where a threshold needs to >>>>> be set to turn this into high-low digital data, then the frames need >>>>> to be found according to start/stop bits and the result output as >>>>> ASCII. >>>>> >>>>> Similar to morse code decoding, just that instead of morse code it's >>>>> RS232 data. Or in my case RS485 but it's the same thing. >>>>> >>>> >>>> "Tomorrow" is a pretty tight deadline I'd have to charge a lot for but >>>> if you can post a set of example data I could whip up a softwares that >>>> might be usable for next time for 'bout $0 >>>> >>> >>> Thanks, but I can't post that publicly. One engineer from the UK wrote >>> me a PM and wants to take a stab at it in Excel. I sent him a recorded >>> CSV file. I don't yet have the data format info, probably one start >>> and one stop bit though I won't know for sure until tomorrow. >>> >>> I can do this one using the old ruler and conting method, hoping I'll >>> never need to do this again for the rest of my life. >>> >>> It just puzzles me that there isn't any software out there for such a >>> decoding. I can't be the only one with an older DSO. >> Sigrok can do it. https://sigrok.org/ >> That software supports cheap USB logic analysers (I know you don't have >> one, so no good for this time, but you should spend the $6.86 so that >> you have one ready for next time): >> https://www.dx.com/p/logic-analyzer-w-dupont-lines-and-usb-cable-for-scm-black-2017429 >> >> >> I understand that Sigrok also supports many DSOs, and may support the >> one that you have, or some intermediate format that you can save to. >> >> It is a while since I installed Sigrok (on linux). I seem to recall >> needing a fairly recent version of linux and Sigrok, though I didn't >> have to build anything from source. It was a pain to set up, but saved >> me a lot of time in the end. I think I used a nightly build appimage: >> https://sigrok.org/jenkins/job/sigrok-native-appimage/platform=native-i686-appimage/lastSuccessfulBuild/artifact/cross-compile/appimage/out/PulseView-NIGHTLY-i686.AppImage >> >> >> > > Thanks. At first glance it looks like it does RS232 only via a Prolific > chip which wouldn't allow my measurements because I have to trigger on > an event: > > https://sigrok.org/wiki/Protocol_decoder:Uart > > IOW, I can't just hook a RS232-USB converter to a PC and log. Unless I > somehow hardwire the scope's trigger output to a pass device that lets > the stream start only after a trigger event. >It appears that you have access to the real-time hardware data. It appears that you're triggering somehow on something. It's not at all clear whether you need to see data prior to the trigger event. Not clear whether this is a one-time measurement or whether you're looking for an infrequent event/bug. If this is a blip on otherwise equal data streams, you might be able to compare a stored good waveform with a bad one. Then you only have to decode the mis-compares. If you have serial data, it's going somewhere. Why can't you use the destination to acquire the data you need? Stated another way, sometimes, focusing on a particular plan of attack, tunnel vision, can prevent examination of far easier paths. Depending on your scope, you may be able to put it into single sweep mode with a LONG sweep time and use the sweep gate out as your enabler. You may not have to gate the data at all. You just have to embed the event in the data. Can you trash a character and detect the parity error? The devil is in the details. Doubt this helps, but I've acquired data using an IR led on the data line and acquired it with the IR port on a PALM III with a serial data app. Use a second LED to saturate/'blind' the acquisition IR until the trigger event. A serial to Bluetooth dongle can make it work on more recent devices without hardware serial or IR. A single transistor into a real RS232 port on an old laptop will reliably acquire RS232 with 0 to 5V swing. Maybe a second transistor depending on your source polarity.
Reply by ●January 21, 20192019-01-21
On Monday, January 21, 2019 at 1:03:25 PM UTC-5, Joerg wrote:> On 2019-01-21 07:51, gnuarm.deletethisbit@gmail.com wrote: > > On Monday, January 21, 2019 at 10:44:34 AM UTC-5, Joerg wrote: > >> On 2019-01-20 16:05, Tim Williams wrote: > >>> https://softwareengineering.stackexchange.com/questions/153351/is-there-an-specific-way-or-algorithm-to-decode-protocols > >>> > >>> > >>> > ? > >>> > >>> Tons of examples of software decode for embedded platforms, might > >>> take longer to refactor for a data series than to write one new > >>> though? > >>> > >>> I'd gladly write one say in JavaScript and put it up on my > >>> website for all to use, but I don't need one, and I'm guessing > >>> you don't have the time/budget to do that on request. > >>> > >>> Well, such is life :) > >>> > >> > >> Yeah, time is the issue and I am now to the point where I pretty > >> much have to decode by hand. > > > > Not really. If you send me a file I will write a program to pull out > > the data. What format do you want the output? > > > > Ideally in hex. I had sent you the file this morning. Meantime Martin > and Lasse sent me code (Martin in Excel VBA with macros, Lasse in C) > which seems to work. They result in the same hex output which is > encouraging. > > Embarrassed to say but I am not familiar with compiling stuff so I'll > probably try to plow ahead with Excel, or in my case try to run it in > OpenOffice.I work in Forth. In theory I could send you a stand alone executable but I'm not familiar with the process of generating that. The other alternative is to install a Forth app to run the code, which would be a bit of effort for you, but not tons. The Forth I use, Win32Forth can sometimes be flagged by AVS even though it is clean. Gforth is fine and I haven't heard of any AVS problems, but I haven't used it under Windows. So do you still need this or is Martin and Lasse's work good enough? Rick C. -- Get 6 months of free supercharging -- Tesla referral code - https://ts.la/richard11209
Reply by ●January 21, 20192019-01-21
On 2019-01-21 12:11, Mike wrote:> On 1/21/2019 7:36 AM, Joerg wrote: >> On 2019-01-21 05:02, Chris Jones wrote: >>> On 21/01/2019 10:06, Joerg wrote: >>>> On 2019-01-20 14:49, bitrex wrote: >>>>> On 01/20/2019 05:26 PM, Joerg wrote: >>>>>> On 2019-01-20 13:54, Sjouke Burry wrote: >>>>>>> On 20-1-2019 22:51, Sjouke Burry wrote: >>>>>>>> On 20-1-2019 21:52, Joerg wrote: >>>>>>>>> Is there a simple Windows or Linux program that can decode serial >>>>>>>>> data >>>>>>>>> out of raw data from an digital oscilloscope? >>>>>>>>> >>>>>>>>> My old logic analyzer does not have any serial data decoding and >>>>>>>>> neither >>>>>>>>> does my scope. So I'd need something that can take in the ADC raw >>>>>>>>> data, >>>>>>>>> I set a threshold to turn that into H/L digital values and the >>>>>>>>> software >>>>>>>>> would fish out the data words. Not sure if it even exist. Not >>>>>>>>> something >>>>>>>>> expensive because I'll only need it once for a short time, to >>>>>>>>> find if >>>>>>>>> there's a bug visible in the data. It'll be a challenge because my >>>>>>>>> scope >>>>>>>>> only has 25,000 points recording length. >>>>>>>>> >>>>>>>> For reading/writing the hardware ports, (for me in DOS) you >>>>>>>> can download my code. >>>>>>>> The exe can be loaded on both sides of a nul modem connection, >>>>>>>> and gives both screen access to the oher computer. >>>>>>>> >>>>>>>> It uses the hardware buffer in the rs232 chip, no interrupt used. >>>>>>>> >>>>>>> Sorry, I forgot the link.> >>>>>>> http://home.planet.nl/~burry004/COMPORT.ZIP >>>>>>> >>>>>> >>>>>> Thanks, Sjouke, but unfortunately this isn't coming from the harware >>>>>> port. It is raw data in the form of an oscilloscope data dump. >>>>>> Basically 25,000 data words of 8-bits each where a threshold needs to >>>>>> be set to turn this into high-low digital data, then the frames need >>>>>> to be found according to start/stop bits and the result output as >>>>>> ASCII. >>>>>> >>>>>> Similar to morse code decoding, just that instead of morse code it's >>>>>> RS232 data. Or in my case RS485 but it's the same thing. >>>>>> >>>>> >>>>> "Tomorrow" is a pretty tight deadline I'd have to charge a lot for but >>>>> if you can post a set of example data I could whip up a softwares that >>>>> might be usable for next time for 'bout $0 >>>>> >>>> >>>> Thanks, but I can't post that publicly. One engineer from the UK wrote >>>> me a PM and wants to take a stab at it in Excel. I sent him a recorded >>>> CSV file. I don't yet have the data format info, probably one start >>>> and one stop bit though I won't know for sure until tomorrow. >>>> >>>> I can do this one using the old ruler and conting method, hoping I'll >>>> never need to do this again for the rest of my life. >>>> >>>> It just puzzles me that there isn't any software out there for such a >>>> decoding. I can't be the only one with an older DSO. >>> Sigrok can do it. https://sigrok.org/ >>> That software supports cheap USB logic analysers (I know you don't have >>> one, so no good for this time, but you should spend the $6.86 so that >>> you have one ready for next time): >>> https://www.dx.com/p/logic-analyzer-w-dupont-lines-and-usb-cable-for-scm-black-2017429 >>> >>> >>> I understand that Sigrok also supports many DSOs, and may support the >>> one that you have, or some intermediate format that you can save to. >>> >>> It is a while since I installed Sigrok (on linux). I seem to recall >>> needing a fairly recent version of linux and Sigrok, though I didn't >>> have to build anything from source. It was a pain to set up, but saved >>> me a lot of time in the end. I think I used a nightly build appimage: >>> https://sigrok.org/jenkins/job/sigrok-native-appimage/platform=native-i686-appimage/lastSuccessfulBuild/artifact/cross-compile/appimage/out/PulseView-NIGHTLY-i686.AppImage >>> >>> >>> >> >> Thanks. At first glance it looks like it does RS232 only via a >> Prolific chip which wouldn't allow my measurements because I have to >> trigger on an event: >> >> https://sigrok.org/wiki/Protocol_decoder:Uart >> >> IOW, I can't just hook a RS232-USB converter to a PC and log. Unless I >> somehow hardwire the scope's trigger output to a pass device that lets >> the stream start only after a trigger event. >> > It appears that you have access to the real-time hardware data. > It appears that you're triggering somehow on something. > It's not at all clear whether you need to see data prior to the trigger > event.No pre-trigger view needed but pre-trigger logging would be easy with my DSO.> Not clear whether this is a one-time measurement or whether you're looking > for an infrequent event/bug. >There is a repetitive bug and some units and on others there isn't.> If this is a blip on otherwise equal data streams, you might be > able to compare a stored good waveform with a bad one. Then you only > have to decode the mis-compares. >It's not that easy. Later in the message there are serial numbers and stuff embedded in the data and that's different between faulty units and non-faulty ones. This makes raw data comaring really tough. In decoded hex it's not so tough.> If you have serial data, it's going somewhere. Why can't you use the > destination to acquire the data you need? >It needs to be acquire at or around an event and that's best done using a DSO or a logic analyzer. My logic analyzer is a model dating almost back to Fred Flintstone when all buses were parallel. So no serial decode there. It also doesn't have fancy trigger such as runt pulse like the DSO does.> Stated another way, sometimes, focusing on a particular plan of attack, > tunnel vision, can prevent examination of far easier paths. > > Depending on your scope, you may be able to put it into single sweep mode > with a LONG sweep time and use the sweep gate out as your enabler. > > You may not have to gate the data at all. You just have to embed the > event in the data. Can you trash a character and detect the parity error? >It's not that easy. This scope only has 25,000 recording length and even that only if limiting to single-channel.> The devil is in the details. >It sure is :-)> Doubt this helps, but I've acquired data using an IR led on the data line > and acquired it with the IR port on a PALM III with a serial data app. > Use a second LED to saturate/'blind' the acquisition IR until the > trigger event. A serial to Bluetooth dongle can make it work > on more recent devices without hardware serial or IR. >That is nifty but if wouldn't allow trigger on a fault.> A single transistor into a real RS232 port on an old laptop will reliably > acquire RS232 with 0 to 5V swing. Maybe a second transistor depending > on your source polarity.xfer of 3.3V logic into RS232 is easy but that won't allow any trigger capture. -- Regards, Joerg http://www.analogconsultants.com/
Reply by ●January 21, 20192019-01-21
On 2019-01-21 12:15, gnuarm.deletethisbit@gmail.com wrote:> On Monday, January 21, 2019 at 1:03:25 PM UTC-5, Joerg wrote: >> On 2019-01-21 07:51, gnuarm.deletethisbit@gmail.com wrote: >>> On Monday, January 21, 2019 at 10:44:34 AM UTC-5, Joerg wrote: >>>> On 2019-01-20 16:05, Tim Williams wrote: >>>>> https://softwareengineering.stackexchange.com/questions/153351/is-there-an-specific-way-or-algorithm-to-decode-protocols >>>>> >>>>> >>>>> >> >>>>>?>>>>> >>>>> Tons of examples of software decode for embedded platforms, >>>>> might take longer to refactor for a data series than to write >>>>> one new though? >>>>> >>>>> I'd gladly write one say in JavaScript and put it up on my >>>>> website for all to use, but I don't need one, and I'm >>>>> guessing you don't have the time/budget to do that on >>>>> request. >>>>> >>>>> Well, such is life :) >>>>> >>>> >>>> Yeah, time is the issue and I am now to the point where I >>>> pretty much have to decode by hand. >>> >>> Not really. If you send me a file I will write a program to pull >>> out the data. What format do you want the output? >>> >> >> Ideally in hex. I had sent you the file this morning. Meantime >> Martin and Lasse sent me code (Martin in Excel VBA with macros, >> Lasse in C) which seems to work. They result in the same hex output >> which is encouraging. >> >> Embarrassed to say but I am not familiar with compiling stuff so >> I'll probably try to plow ahead with Excel, or in my case try to >> run it in OpenOffice. > > I work in Forth. In theory I could send you a stand alone executable > but I'm not familiar with the process of generating that. The other > alternative is to install a Forth app to run the code, which would be > a bit of effort for you, but not tons. The Forth I use, Win32Forth > can sometimes be flagged by AVS even though it is clean. Gforth is > fine and I haven't heard of any AVS problems, but I haven't used it > under Windows. > > So do you still need this or is Martin and Lasse's work good enough? >Thanks, but it should definitely be good enough. Currently the code in the uC firmware is being looked at because a bug is now suspected in it, which could be causing all this. -- Regards, Joerg http://www.analogconsultants.com/
Reply by ●January 21, 20192019-01-21
On 2019-01-21 07:45, John Larkin wrote:> On Mon, 21 Jan 2019 07:17:35 -0800, Joerg <news@analogconsultants.com> > wrote: > >> On 2019-01-20 15:00, John Larkin wrote: >>> On Sun, 20 Jan 2019 13:43:12 -0800, Joerg <news@analogconsultants.com> >>> wrote: >>> >>>> On 2019-01-20 13:21, Lasse Langwadt Christensen wrote: >>>>> s�ndag den 20. januar 2019 kl. 21.52.53 UTC+1 skrev Joerg: >>>>>> Is there a simple Windows or Linux program that can decode serial >>>>>> data out of raw data from an digital oscilloscope? >>>>>> >>>>>> My old logic analyzer does not have any serial data decoding and >>>>>> neither does my scope. So I'd need something that can take in the >>>>>> ADC raw data, I set a threshold to turn that into H/L digital >>>>>> values and the software would fish out the data words. Not sure if >>>>>> it even exist. Not something expensive because I'll only need it >>>>>> once for a short time, to find if there's a bug visible in the >>>>>> data. It'll be a challenge because my scope only has 25,000 points >>>>>> recording length. >>>>> >>>>> get a cheap logic usb analyser? >>>>> https://learn.sparkfun.com/tutorials/using-the-usb-logic-analyzer-with-sigrok-pulseview/all >>>>> >>>> >>>> That sure would ne nice but I have to do this tomorrow. >>>> >>>> >>>>> else it would take long to program, just slice that data into 1/0 and >>>>> grap/decode a frame worth of data after each start bit >>>>> >>>> >>>> I thought there had to be some programs that already can do that. Many >>>> times people sit there and decode by hand. I used to do that in my olden >>>> days, using a ruler and stacks of Polaroid pictures from a scope screen. >>>> >>>> It should even be possible to do this in an Excel macro and then load in >>> >from the comma-delimited data file but I am not that great a programmer, >>>> I am an analog guy. >>>> >>>> >>>>> what's the format of the data from the scope? >>>>> >>>> >>>> It can store raw CSV for the data. Also BMP, JPEG and some others but >>>> those only per screen and not the whole recording. >>>> >>>> If there is nothing I'll just use the old print-out, ruler and counting >>>> method. >>> >>> Why not connect your RS232 to a serial port on a PC or laptop? >>> >> >> It comes in the form of logic level inside a circuit. I can try to whip >> up some sauerkraut electronics to transfer that to bipolar RS232. One of >> the wee challenges would be to sync two streams, TX and RX. With a scope >> that's easy, I can just trigger on an event on the TX line and record >> what RX responds with. > > Most RS232 receivers work fine with TTL input (unless it's inverted!) >I've got 3.3V but easy to handle using a transistor. The problem is the event sync. My scope doesn't have a trigger-out, only a "good-bad" qualifier output that can turn on a the big red light but that's not 100% realtime. -- Regards, Joerg http://www.analogconsultants.com/
Reply by ●January 21, 20192019-01-21
On 01/21/2019 01:03 PM, Joerg wrote:> On 2019-01-21 07:51, gnuarm.deletethisbit@gmail.com wrote: >> On Monday, January 21, 2019 at 10:44:34 AM UTC-5, Joerg wrote: >>> On 2019-01-20 16:05, Tim Williams wrote: >>>> https://softwareengineering.stackexchange.com/questions/153351/is-there-an-specific-way-or-algorithm-to-decode-protocols >>>> >>>> >>>> >>>> > ? >>>> >>>> Tons of examples of software decode for embedded platforms, might >>>> take longer to refactor for a data series than to write one new >>>> though? >>>> >>>> I'd gladly write one say in JavaScript and put it up on my >>>> website for all to use, but I don't need one, and I'm guessing >>>> you don't have the time/budget to do that on request. >>>> >>>> Well, such is life :) >>>> >>> >>> Yeah, time is the issue and I am now to the point where I pretty >>> much have to decode by hand. >> >> Not really. If you send me a file I will write a program to pull out >> the data. What format do you want the output? >> > > Ideally in hex. I had sent you the file this morning. Meantime Martin > and Lasse sent me code (Martin in Excel VBA with macros, Lasse in C) > which seems to work. They result in the same hex output which is > encouraging. > > Embarrassed to say but I am not familiar with compiling stuff so I'll > probably try to plow ahead with Excel, or in my case try to run it in > OpenOffice. >Gosh that's quite nice of Lasse, I'd have least asked for some beer money to do it in C! Dealing with large numbers of data points from a data set of arbitrary size is a lot more civilized in C++ with its dynamically-re sizable containers (std::list, std::vector, etc.)
Reply by ●January 21, 20192019-01-21
On 01/21/2019 06:21 PM, bitrex wrote:> On 01/21/2019 01:03 PM, Joerg wrote: >> On 2019-01-21 07:51, gnuarm.deletethisbit@gmail.com wrote: >>> On Monday, January 21, 2019 at 10:44:34 AM UTC-5, Joerg wrote: >>>> On 2019-01-20 16:05, Tim Williams wrote: >>>>> https://softwareengineering.stackexchange.com/questions/153351/is-there-an-specific-way-or-algorithm-to-decode-protocols >>>>> >>>>> >>>>> >>>>> >> ? >>>>> >>>>> Tons of examples of software decode for embedded platforms, might >>>>> take longer to refactor for a data series than to write one new >>>>> though? >>>>> >>>>> I'd gladly write one say in JavaScript and put it up on my >>>>> website for all to use, but I don't need one, and I'm guessing >>>>> you don't have the time/budget to do that on request. >>>>> >>>>> Well, such is life :) >>>>> >>>> >>>> Yeah, time is the issue and I am now to the point where I pretty >>>> much have to decode by hand. >>> >>> Not really. If you send me a file I will write a program to pull out >>> the data. What format do you want the output? >>> >> >> Ideally in hex. I had sent you the file this morning. Meantime Martin >> and Lasse sent me code (Martin in Excel VBA with macros, Lasse in C) >> which seems to work. They result in the same hex output which is >> encouraging. >> >> Embarrassed to say but I am not familiar with compiling stuff so I'll >> probably try to plow ahead with Excel, or in my case try to run it in >> OpenOffice. >> > > Gosh that's quite nice of Lasse, I'd have least asked for some beer > money to do it in C! > > Dealing with large numbers of data points from a data set of arbitrary > size is a lot more civilized in C++ with its dynamically-re sizable > containers (std::list, std::vector, etc.)Girlfriend had a bad case of the sniffles and I know I'd say "Take a Sudafed and deal, sweetie, there's work to be done" for some price. Less than a $1 million. More than $0. I might have to put more thought into the relationship-Laffer-curve at some point
Reply by ●January 21, 20192019-01-21
tirsdag den 22. januar 2019 kl. 00.21.55 UTC+1 skrev bitrex:> On 01/21/2019 01:03 PM, Joerg wrote: > > On 2019-01-21 07:51, gnuarm.deletethisbit@gmail.com wrote: > >> On Monday, January 21, 2019 at 10:44:34 AM UTC-5, Joerg wrote: > >>> On 2019-01-20 16:05, Tim Williams wrote: > >>>> https://softwareengineering.stackexchange.com/questions/153351/is-there-an-specific-way-or-algorithm-to-decode-protocols > >>>> > >>>> > >>>> > >>>> > > ? > >>>> > >>>> Tons of examples of software decode for embedded platforms, might > >>>> take longer to refactor for a data series than to write one new > >>>> though? > >>>> > >>>> I'd gladly write one say in JavaScript and put it up on my > >>>> website for all to use, but I don't need one, and I'm guessing > >>>> you don't have the time/budget to do that on request. > >>>> > >>>> Well, such is life :) > >>>> > >>> > >>> Yeah, time is the issue and I am now to the point where I pretty > >>> much have to decode by hand. > >> > >> Not really. If you send me a file I will write a program to pull out > >> the data. What format do you want the output? > >> > > > > Ideally in hex. I had sent you the file this morning. Meantime Martin > > and Lasse sent me code (Martin in Excel VBA with macros, Lasse in C) > > which seems to work. They result in the same hex output which is > > encouraging. > > > > Embarrassed to say but I am not familiar with compiling stuff so I'll > > probably try to plow ahead with Excel, or in my case try to run it in > > OpenOffice. > > > > Gosh that's quite nice of Lasse, I'd have least asked for some beer > money to do it in C! > > Dealing with large numbers of data points from a data set of arbitrary > size is a lot more civilized in C++ with its dynamically-re sizable > containers (std::list, std::vector, etc.)the data size was know at compile time and no matter what language you use I think you'd want to figure out how much space you need and allocate that at the beginning so you don't have a vector grow as you read numbers from a file
Reply by ●January 21, 20192019-01-21
On 01/21/2019 07:01 PM, Lasse Langwadt Christensen wrote:> tirsdag den 22. januar 2019 kl. 00.21.55 UTC+1 skrev bitrex: >> On 01/21/2019 01:03 PM, Joerg wrote: >>> On 2019-01-21 07:51, gnuarm.deletethisbit@gmail.com wrote: >>>> On Monday, January 21, 2019 at 10:44:34 AM UTC-5, Joerg wrote: >>>>> On 2019-01-20 16:05, Tim Williams wrote: >>>>>> https://softwareengineering.stackexchange.com/questions/153351/is-there-an-specific-way-or-algorithm-to-decode-protocols >>>>>> >>>>>> >>>>>> >>>>>> >>> ? >>>>>> >>>>>> Tons of examples of software decode for embedded platforms, might >>>>>> take longer to refactor for a data series than to write one new >>>>>> though? >>>>>> >>>>>> I'd gladly write one say in JavaScript and put it up on my >>>>>> website for all to use, but I don't need one, and I'm guessing >>>>>> you don't have the time/budget to do that on request. >>>>>> >>>>>> Well, such is life :) >>>>>> >>>>> >>>>> Yeah, time is the issue and I am now to the point where I pretty >>>>> much have to decode by hand. >>>> >>>> Not really. If you send me a file I will write a program to pull out >>>> the data. What format do you want the output? >>>> >>> >>> Ideally in hex. I had sent you the file this morning. Meantime Martin >>> and Lasse sent me code (Martin in Excel VBA with macros, Lasse in C) >>> which seems to work. They result in the same hex output which is >>> encouraging. >>> >>> Embarrassed to say but I am not familiar with compiling stuff so I'll >>> probably try to plow ahead with Excel, or in my case try to run it in >>> OpenOffice. >>> >> >> Gosh that's quite nice of Lasse, I'd have least asked for some beer >> money to do it in C! >> >> Dealing with large numbers of data points from a data set of arbitrary >> size is a lot more civilized in C++ with its dynamically-re sizable >> containers (std::list, std::vector, etc.) > > the data size was know at compile time and no matter what language you > use I think you'd want to figure out how much space you need and allocate > that at the beginning so you don't have a vector grow as you read numbers > from a file >For record sizes of tens of thousands of byte-sized elements or CSVs I suppose on modern PCs it hardly matters exactly how the data is loaded and processed it'll all be done before you know it. vectors are very generalized data structures, they can be allocated on the stack or heap, the stack is usually a couple megabytes in size on most modern desktop arch so for 25,0000 byte-sized records probably can just use the stack no heap allocation required at all. The overhead of simply opening, reading, and closing the file dominates the runtime by a longshot. in that case it's mostly a matter of convenience from a programming/algorithm implementation perspective, once the data is in a generic container like a vector, however it got there, it's easy to write a std::algorithm of some kind that operates on the entire container in some way automatically, an element-wise mapping, in-place or by returning values, however you like. I don't like "naked" for loops and avoid them when I can, they're error-prone the "modern" solution is to use iterators and avoid touching your actual data directly when you don't have to. For large record sizes like millions or tens of millions of elements, where the amount of data is approaching a goodly percentage of the total available heap memory of the machine it makes sense to read the file in piece by piece in manageable chunks like multiples of a cache line size that would fit in an L2 cache, and farm the work out to worker threads leveraging multiple cores.
