Shared Communications Bus - RS-422 or RS-485

On Saturday, December 3, 2022 at 4:16:28 PM UTC-5, lang...@fonz.dk wrote:
> lørdag den 3. december 2022 kl. 21.57.49 UTC+1 skrev Ricky: 
> > On Saturday, December 3, 2022 at 7:00:04 AM UTC-5, upsid...@downunder.com wrote: 
> > > On Sat, 3 Dec 2022 00:10:13 -0800 (PST), Ricky 
> > > <gnuarm.del...@gmail.com> wrote: 
> > > 
> > > > 
> > > >> IMHO the simplest thing to make the connection would to use an 
> > > >> Ethernet pot on a lap/table top, use two 8 port (or four 4 port) 
> > > >> Ethernet to RS-42//485 converters. This you would have 16 independent 
> > > >> multidrop buses. Put the converters in UDP mode and the programming is 
> > > >> easy. Write the whole Tx message into the socket and receive a full 
> > > >> serial frame from the UDP socket. 
> > > > 
> > > >What programming? The serial converters have drivers to make them look like COM ports on the PC. The code to talk to the serial port is already written and working. 
> > > The problem with those vendor supplied COM port look-a-likes is that 
> > > they only support a single line. If you have multiple lines, you will 
> > > have to use a process (or at least a thread) for each physical serial 
> > > line to get any advantage of using multiple lines. 
> > I will look into that. I've not been told I can't open multiple COM ports from a single process.
> you can open as many as you like, they are mostly just like files
> > > Using UDP sockets, you can send a single request to every serial line 
> > > in parallel. Then start waiting for responses from these lines, by 
> > > issuing a select() call and process each response as it has arrived. 
> > > This can all be done un a single thread. I have used this technique to 
> > > drive up to 50 serial lines with Eth/serial converters controlling 50 
> > > Variable Frequency Drives (VFD) running 50 motors from a single 
> > > thread. 
> > How does the UDP socket map to a serial port? What driver is required from the adapter vendor? 
> > 
> > I'm not looking to do tons of work for this. Actually, I finally got past my initial resistance to an idea that might allow high data rates on a single COM port, as well as controlling the timing of the slaves on the shared bus. DEL characters can be added to the master messages to space them out. Since the slaves respond very rapidly, the spacing will give the first slave time to complete the reply before the second slave starts sending. Since the master sends large messages and the slave replies should look like a single message, this should be very fast in the end, with a single serial port. 
> >
> yes, concat your messages spaced with some special character to give each slave a slot and send that as one long message

Yeah, that will work, as long as the slave messages will be "seen" as a single message and not a bunch of little messages.  That's where all the trouble is coming from, each message being treated as a separate entity to be shipped through the adapter.  

These will show up oddly in a debugger.  

One vendor suggested I use their product and customize the software myself.  It runs under linux.  I think I would prefer to build a separate unit, that sits between the serial port and the test fixture bus, separating the commands and waiting for the reply.  Actually, I could incorporate this into the test fixture.  So the first board receives the data stream from the PC through a separate port, and sends the commands, one a a time, with a bit of space.  No need for idle characters to be sent.  

I'll have a couple of weeks to give this thought while I wait for the serial adapter from FTDI.  

-- 

Rick C.

---+- Get 1,000 miles of free Supercharging
---+- Tesla referral code - https://ts.la/richard11209

On 2022-12-03, Ricky <gnuarm.deletethisbit@gmail.com> wrote:
> On Saturday, December 3, 2022 at 7:00:04 AM UTC-5, upsid...@downunder.com wrote:
>> On Sat, 3 Dec 2022 00:10:13 -0800 (PST), Ricky 
>> <gnuarm.del...@gmail.com> wrote: 
>> 
>> > 
>> >> IMHO the simplest thing to make the connection would to use an 
>> >> Ethernet pot on a lap/table top, use two 8 port (or four 4 port) 
>> >> Ethernet to RS-42//485 converters. This you would have 16 independent 
>> >> multidrop buses. Put the converters in UDP mode and the programming is 
>> >> easy. Write the whole Tx message into the socket and receive a full 
>> >> serial frame from the UDP socket. 
>> > 
>> >What programming? The serial converters have drivers to make them look like COM ports on the PC. The code to talk to the serial port is already written and working.
>> The problem with those vendor supplied COM port look-a-likes is that 
>> they only support a single line. If you have multiple lines, you will 
>> have to use a process (or at least a thread) for each physical serial 
>> line to get any advantage of using multiple lines. 
>
> I will look into that.  I've not been told I can't open multiple COM ports from a single process. 
>
>
>> Using UDP sockets, you can send a single request to every serial line 
>> in parallel. Then start waiting for responses from these lines, by 
>> issuing a select() call and process each response as it has arrived. 
>> This can all be done un a single thread. I have used this technique to 
>> drive up to 50 serial lines with Eth/serial converters controlling 50 
>> Variable Frequency Drives (VFD) running 50 motors from a single 
>> thread.
>
> How does the UDP socket map to a serial port?  What driver is required from the adapter vendor?  

On unix both are regular IO file handles and resposnd to the same sort of
manipulation using poll() and select() etc.  basically all you need to
do to convert your code from UART to UDP is swap the open() call for a
connect() call.

On windows the UDP socket is a network handle and supports poll and select
but the com port is a special class of file handle and there are different
calls to do everything.

OSI layers 4 and down basically map UDP packets directly to ethernet datagrams 
(just adding a little wrapper)

> I'm not looking to do tons of work for this.  Actually, I finally
>got past my initial resistance to an idea that might allow high data
>rates on a single COM port, as well as controlling the timing of the
>slaves on the shared bus.  DEL characters can be added to the master
>messages to space them out.  Since the slaves respond very rapidly,
>the spacing will give the first slave time to complete the reply 
>before the second slave starts sending.   Since the master sends 
>large messages and the slave replies should look like a single 
>message, this should be very fast in the end, with a single 
>serial port.

That seems like it would work. NUL ('\x00') was traditionally
used in that application (eg to insert pauses for mechanical 
TTYs to return the carriage and crank the paper, or for glass 
TTYs to scroll the video ram), but yeah you don't need to use
ASCII control codes as intended.

-- 
  Jasen.

lørdag den 3. december 2022 kl. 23.28.30 UTC+1 skrev Ricky:
> On Saturday, December 3, 2022 at 4:16:28 PM UTC-5, lang...@fonz.dk wrote: 
> > lørdag den 3. december 2022 kl. 21.57.49 UTC+1 skrev Ricky: 
> > > On Saturday, December 3, 2022 at 7:00:04 AM UTC-5, upsid...@downunder.com wrote: 
> > > > On Sat, 3 Dec 2022 00:10:13 -0800 (PST), Ricky 
> > > > <gnuarm.del...@gmail.com> wrote: 
> > > > 
> > > > > 
> > > > >> IMHO the simplest thing to make the connection would to use an 
> > > > >> Ethernet pot on a lap/table top, use two 8 port (or four 4 port) 
> > > > >> Ethernet to RS-42//485 converters. This you would have 16 independent 
> > > > >> multidrop buses. Put the converters in UDP mode and the programming is 
> > > > >> easy. Write the whole Tx message into the socket and receive a full 
> > > > >> serial frame from the UDP socket. 
> > > > > 
> > > > >What programming? The serial converters have drivers to make them look like COM ports on the PC. The code to talk to the serial port is already written and working. 
> > > > The problem with those vendor supplied COM port look-a-likes is that 
> > > > they only support a single line. If you have multiple lines, you will 
> > > > have to use a process (or at least a thread) for each physical serial 
> > > > line to get any advantage of using multiple lines. 
> > > I will look into that. I've not been told I can't open multiple COM ports from a single process. 
> > you can open as many as you like, they are mostly just like files 
> > > > Using UDP sockets, you can send a single request to every serial line 
> > > > in parallel. Then start waiting for responses from these lines, by 
> > > > issuing a select() call and process each response as it has arrived. 
> > > > This can all be done un a single thread. I have used this technique to 
> > > > drive up to 50 serial lines with Eth/serial converters controlling 50 
> > > > Variable Frequency Drives (VFD) running 50 motors from a single 
> > > > thread. 
> > > How does the UDP socket map to a serial port? What driver is required from the adapter vendor? 
> > > 
> > > I'm not looking to do tons of work for this. Actually, I finally got past my initial resistance to an idea that might allow high data rates on a single COM port, as well as controlling the timing of the slaves on the shared bus. DEL characters can be added to the master messages to space them out. Since the slaves respond very rapidly, the spacing will give the first slave time to complete the reply before the second slave starts sending. Since the master sends large messages and the slave replies should look like a single message, this should be very fast in the end, with a single serial port. 
> > > 
> > yes, concat your messages spaced with some special character to give each slave a slot and send that as one long message
> Yeah, that will work, as long as the slave messages will be "seen" as a single message and not a bunch of little messages. That's where all the trouble is coming from, each message being treated as a separate entity to be shipped through the adapter. 
> 

the ftdi will just send data it receives it on the uart side over USB as it fills frames and the rest when some idle period occurred

on the PC side you just read X*Y bytes with a timeout and either you get it or you get a timeout and how many bytes you did get

> These will show up oddly in a debugger. 
> 
> One vendor suggested I use their product and customize the software myself. It runs under linux. I think I would prefer to build a separate unit, that sits between the serial port and the test fixture bus, separating the commands and waiting for the reply. Actually, I could incorporate this into the test fixture. So the first board receives the data stream from the PC through a separate port, and sends the commands, one a a time, with a bit of space. No need for idle characters to be sent. 
> 

 with a MCU it would be very trivial to send/recieve messages with fixed timing, but PC is like a semitruck it can move 
lots of stuff fast but it isn't very good at reacting fast to changes 

On Saturday, December 3, 2022 at 8:22:16 PM UTC-5, lang...@fonz.dk wrote:
> lørdag den 3. december 2022 kl. 23.28.30 UTC+1 skrev Ricky: 
> > On Saturday, December 3, 2022 at 4:16:28 PM UTC-5, lang...@fonz.dk wrote: 
> > > lørdag den 3. december 2022 kl. 21.57.49 UTC+1 skrev Ricky: 
> > > > On Saturday, December 3, 2022 at 7:00:04 AM UTC-5, upsid...@downunder.com wrote: 
> > > > > On Sat, 3 Dec 2022 00:10:13 -0800 (PST), Ricky 
> > > > > <gnuarm.del...@gmail.com> wrote: 
> > > > > 
> > > > > > 
> > > > > >> IMHO the simplest thing to make the connection would to use an 
> > > > > >> Ethernet pot on a lap/table top, use two 8 port (or four 4 port) 
> > > > > >> Ethernet to RS-42//485 converters. This you would have 16 independent 
> > > > > >> multidrop buses. Put the converters in UDP mode and the programming is 
> > > > > >> easy. Write the whole Tx message into the socket and receive a full 
> > > > > >> serial frame from the UDP socket. 
> > > > > > 
> > > > > >What programming? The serial converters have drivers to make them look like COM ports on the PC. The code to talk to the serial port is already written and working. 
> > > > > The problem with those vendor supplied COM port look-a-likes is that 
> > > > > they only support a single line. If you have multiple lines, you will 
> > > > > have to use a process (or at least a thread) for each physical serial 
> > > > > line to get any advantage of using multiple lines. 
> > > > I will look into that. I've not been told I can't open multiple COM ports from a single process. 
> > > you can open as many as you like, they are mostly just like files 
> > > > > Using UDP sockets, you can send a single request to every serial line 
> > > > > in parallel. Then start waiting for responses from these lines, by 
> > > > > issuing a select() call and process each response as it has arrived. 
> > > > > This can all be done un a single thread. I have used this technique to 
> > > > > drive up to 50 serial lines with Eth/serial converters controlling 50 
> > > > > Variable Frequency Drives (VFD) running 50 motors from a single 
> > > > > thread. 
> > > > How does the UDP socket map to a serial port? What driver is required from the adapter vendor? 
> > > > 
> > > > I'm not looking to do tons of work for this. Actually, I finally got past my initial resistance to an idea that might allow high data rates on a single COM port, as well as controlling the timing of the slaves on the shared bus. DEL characters can be added to the master messages to space them out. Since the slaves respond very rapidly, the spacing will give the first slave time to complete the reply before the second slave starts sending. Since the master sends large messages and the slave replies should look like a single message, this should be very fast in the end, with a single serial port. 
> > > > 
> > > yes, concat your messages spaced with some special character to give each slave a slot and send that as one long message 
> > Yeah, that will work, as long as the slave messages will be "seen" as a single message and not a bunch of little messages. That's where all the trouble is coming from, each message being treated as a separate entity to be shipped through the adapter. 
> >
> the ftdi will just send data it receives it on the uart side over USB as it fills frames and the rest when some idle period occurred 
> 
> on the PC side you just read X*Y bytes with a timeout and either you get it or you get a timeout and how many bytes you did get

You aren't following the entire matter.  The problem with sending individual messages is, there are delays, latencies, in handling the data from one side to the other in the adapter.  In the case of the USB device, I'm let to believe this is because of intentional pausing.  They have not indicated the purpose, but in one direction it may be a timeout, so that if no more data is received, the block of data is sent, even if not full (slaves to master).  

In the original approach, both directions would have the delay, because only one message was sent, then the master would wait for the reply.  So the master message (command) would see a wait for a "slot" on the polled USB bus, then the reply would see a similar wait.  By concatenating many commands from the master into one message, the waits are reduced.  But this is only half of the problem.  The slave replies must also be sent together to eliminate the many waits on the reply side.  Since I don't know for sure the nature of the waits, I can't be sure if the short delays between the slave replies will get around the waits. 


> > These will show up oddly in a debugger. 
> > 
> > One vendor suggested I use their product and customize the software myself. It runs under linux. I think I would prefer to build a separate unit, that sits between the serial port and the test fixture bus, separating the commands and waiting for the reply. Actually, I could incorporate this into the test fixture. So the first board receives the data stream from the PC through a separate port, and sends the commands, one a a time, with a bit of space. No need for idle characters to be sent. 
> >
> with a MCU it would be very trivial to send/recieve messages with fixed timing, but PC is like a semitruck it can move 
> lots of stuff fast but it isn't very good at reacting fast to changes

I think the low level drivers have adequate priority to handle the traffic.  At least one vendor has done tests which I believe time the sending on the PC to the receiving on the PC.  So even if the PC makes delays, they are factored in. 

Of course, I can't know for sure exactly how they did the tests.  One thing that reduces the times is they looped back the data, so one transmission time on the serial bus was not included.  But that's not large, so can probably be ignored. 

If the aggregated messages work, that will be the easiest method so far.  I've gotten a lot of advice that is from left field, where people didn't take the time to actually read what I had written before replying.  Free advice, worth the price.  lol 

I really am surprised at the delays in the Ethernet devices.  It never occurred to me they would run all the data for 16 serial ports and the Ethernet bus through a single CPU.  It's surprising it works well at all.  For me, it doesn't. 

About the fast bus not reacting quickly to changes, I've tried to explain to people before, that the Internet isn't just about the "speed" numbers various web sites measure.  If you are watching a movie, you need speed.  If you are browsing the web, you want low latency as well.  No one knows what that means.  I try to explain it's the difference between a road having a high speed limit, but being full of traffic lights.  Most don't really get it still. 

-- 

Rick C.

---++ Get 1,000 miles of free Supercharging
---++ Tesla referral code - https://ts.la/richard11209

søndag den 4. december 2022 kl. 03.35.54 UTC+1 skrev Ricky:
> On Saturday, December 3, 2022 at 8:22:16 PM UTC-5, lang...@fonz.dk wrote: 
> > lørdag den 3. december 2022 kl. 23.28.30 UTC+1 skrev Ricky: 
> > > On Saturday, December 3, 2022 at 4:16:28 PM UTC-5, lang...@fonz.dk wrote: 
> > > > lørdag den 3. december 2022 kl. 21.57.49 UTC+1 skrev Ricky: 
> > > > > On Saturday, December 3, 2022 at 7:00:04 AM UTC-5, upsid...@downunder.com wrote: 
> > > > > > On Sat, 3 Dec 2022 00:10:13 -0800 (PST), Ricky 
> > > > > > <gnuarm.del...@gmail.com> wrote: 
> > > > > > 
> > > > > > > 
> > > > > > >> IMHO the simplest thing to make the connection would to use an 
> > > > > > >> Ethernet pot on a lap/table top, use two 8 port (or four 4 port) 
> > > > > > >> Ethernet to RS-42//485 converters. This you would have 16 independent 
> > > > > > >> multidrop buses. Put the converters in UDP mode and the programming is 
> > > > > > >> easy. Write the whole Tx message into the socket and receive a full 
> > > > > > >> serial frame from the UDP socket. 
> > > > > > > 
> > > > > > >What programming? The serial converters have drivers to make them look like COM ports on the PC. The code to talk to the serial port is already written and working. 
> > > > > > The problem with those vendor supplied COM port look-a-likes is that 
> > > > > > they only support a single line. If you have multiple lines, you will 
> > > > > > have to use a process (or at least a thread) for each physical serial 
> > > > > > line to get any advantage of using multiple lines. 
> > > > > I will look into that. I've not been told I can't open multiple COM ports from a single process. 
> > > > you can open as many as you like, they are mostly just like files 
> > > > > > Using UDP sockets, you can send a single request to every serial line 
> > > > > > in parallel. Then start waiting for responses from these lines, by 
> > > > > > issuing a select() call and process each response as it has arrived. 
> > > > > > This can all be done un a single thread. I have used this technique to 
> > > > > > drive up to 50 serial lines with Eth/serial converters controlling 50 
> > > > > > Variable Frequency Drives (VFD) running 50 motors from a single 
> > > > > > thread. 
> > > > > How does the UDP socket map to a serial port? What driver is required from the adapter vendor? 
> > > > > 
> > > > > I'm not looking to do tons of work for this. Actually, I finally got past my initial resistance to an idea that might allow high data rates on a single COM port, as well as controlling the timing of the slaves on the shared bus. DEL characters can be added to the master messages to space them out. Since the slaves respond very rapidly, the spacing will give the first slave time to complete the reply before the second slave starts sending. Since the master sends large messages and the slave replies should look like a single message, this should be very fast in the end, with a single serial port. 
> > > > > 
> > > > yes, concat your messages spaced with some special character to give each slave a slot and send that as one long message 
> > > Yeah, that will work, as long as the slave messages will be "seen" as a single message and not a bunch of little messages. That's where all the trouble is coming from, each message being treated as a separate entity to be shipped through the adapter. 
> > > 
> > the ftdi will just send data it receives it on the uart side over USB as it fills frames and the rest when some idle period occurred 
> > 
> > on the PC side you just read X*Y bytes with a timeout and either you get it or you get a timeout and how many bytes you did get
> You aren't following the entire matter. 

I do believe I am ... you want to do ping-pong messaging with a microsecond latencies and that is just not going to happen with a PC unless you use a PCI card and write a custom kernel driver

you have to "pipeline" it so you send a big message and get a big reply

>The problem with sending individual messages is, there are delays, latencies, in handling the data from one side to the other in the adapter. In the case of the USB device, I'm let to believe this is because of intentional pausing. They have not indicated the purpose, but in one direction it may be a timeout, so that if no more data is received, the block of data is sent, even if not full (slaves to master). 
> 
> In the original approach, both directions would have the delay, because only one message was sent, then the master would wait for the reply. So the master message (command) would see a wait for a "slot" on the polled USB bus, then the reply would see a similar wait. By concatenating many commands from the master into one message, the waits are reduced. But this is only half of the problem. The slave replies must also be sent together to eliminate the many waits on the reply side. Since I don't know for sure the nature of the waits, I can't be sure if the short delays between the slave replies will get around the waits.

the replies gets buffered in the FTDI, if the slaves all reply before the next slot you get them all

is it an issue that it takes a few frames, when it is for all the slave responses and not per slave response ?

> > > These will show up oddly in a debugger. 
> > > 
> > > One vendor suggested I use their product and customize the software myself. It runs under linux. I think I would prefer to build a separate unit, that sits between the serial port and the test fixture bus, separating the commands and waiting for the reply. Actually, I could incorporate this into the test fixture. So the first board receives the data stream from the PC through a separate port, and sends the commands, one a a time, with a bit of space. No need for idle characters to be sent. 
> > > 
> > with a MCU it would be very trivial to send/recieve messages with fixed timing, but PC is like a semitruck it can move 
> > lots of stuff fast but it isn't very good at reacting fast to changes
> I think the low level drivers have adequate priority to handle the traffic. At least one vendor has done tests which I believe time the sending on the PC to the receiving on the PC. So even if the PC makes delays, they are factored in. 
> 
> Of course, I can't know for sure exactly how they did the tests. One thing that reduces the times is they looped back the data, so one transmission time on the serial bus was not included. But that's not large, so can probably be ignored. 
> 
> If the aggregated messages work, that will be the easiest method so far. I've gotten a lot of advice that is from left field, where people didn't take the time to actually read what I had written before replying. Free advice, worth the price. lol 
> 
> I really am surprised at the delays in the Ethernet devices. It never occurred to me they would run all the data for 16 serial ports and the Ethernet bus through a single CPU. It's surprising it works well at all. For me, it doesn't. 
> 
> About the fast bus not reacting quickly to changes, I've tried to explain to people before, that the Internet isn't just about the "speed" numbers various web sites measure. If you are watching a movie, you need speed. If you are browsing the web, you want low latency as well. No one knows what that means. I try to explain it's the difference between a road having a high speed limit, but being full of traffic lights. Most don't really get it still. 
> 

watching a movie or browsing the web doesn't need that low latency,  you won't notice latency until it gets close to several 10s of milliseconds ,  a 50Hz monitor is 20ms

On Saturday, December 3, 2022 at 10:21:39 PM UTC-5, lang...@fonz.dk wrote:
> søndag den 4. december 2022 kl. 03.35.54 UTC+1 skrev Ricky: 
> > On Saturday, December 3, 2022 at 8:22:16 PM UTC-5, lang...@fonz.dk wrote: 
> > > lørdag den 3. december 2022 kl. 23.28.30 UTC+1 skrev Ricky: 
> > > > On Saturday, December 3, 2022 at 4:16:28 PM UTC-5, lang...@fonz.dk wrote: 
> > > > > lørdag den 3. december 2022 kl. 21.57.49 UTC+1 skrev Ricky: 
> > > > > > On Saturday, December 3, 2022 at 7:00:04 AM UTC-5, upsid...@downunder.com wrote: 
> > > > > > > On Sat, 3 Dec 2022 00:10:13 -0800 (PST), Ricky 
> > > > > > > <gnuarm.del...@gmail.com> wrote: 
> > > > > > > 
> > > > > > > > 
> > > > > > > >> IMHO the simplest thing to make the connection would to use an 
> > > > > > > >> Ethernet pot on a lap/table top, use two 8 port (or four 4 port) 
> > > > > > > >> Ethernet to RS-42//485 converters. This you would have 16 independent 
> > > > > > > >> multidrop buses. Put the converters in UDP mode and the programming is 
> > > > > > > >> easy. Write the whole Tx message into the socket and receive a full 
> > > > > > > >> serial frame from the UDP socket. 
> > > > > > > > 
> > > > > > > >What programming? The serial converters have drivers to make them look like COM ports on the PC. The code to talk to the serial port is already written and working. 
> > > > > > > The problem with those vendor supplied COM port look-a-likes is that 
> > > > > > > they only support a single line. If you have multiple lines, you will 
> > > > > > > have to use a process (or at least a thread) for each physical serial 
> > > > > > > line to get any advantage of using multiple lines. 
> > > > > > I will look into that. I've not been told I can't open multiple COM ports from a single process. 
> > > > > you can open as many as you like, they are mostly just like files 
> > > > > > > Using UDP sockets, you can send a single request to every serial line 
> > > > > > > in parallel. Then start waiting for responses from these lines, by 
> > > > > > > issuing a select() call and process each response as it has arrived. 
> > > > > > > This can all be done un a single thread. I have used this technique to 
> > > > > > > drive up to 50 serial lines with Eth/serial converters controlling 50 
> > > > > > > Variable Frequency Drives (VFD) running 50 motors from a single 
> > > > > > > thread. 
> > > > > > How does the UDP socket map to a serial port? What driver is required from the adapter vendor? 
> > > > > > 
> > > > > > I'm not looking to do tons of work for this. Actually, I finally got past my initial resistance to an idea that might allow high data rates on a single COM port, as well as controlling the timing of the slaves on the shared bus. DEL characters can be added to the master messages to space them out. Since the slaves respond very rapidly, the spacing will give the first slave time to complete the reply before the second slave starts sending. Since the master sends large messages and the slave replies should look like a single message, this should be very fast in the end, with a single serial port. 
> > > > > > 
> > > > > yes, concat your messages spaced with some special character to give each slave a slot and send that as one long message 
> > > > Yeah, that will work, as long as the slave messages will be "seen" as a single message and not a bunch of little messages. That's where all the trouble is coming from, each message being treated as a separate entity to be shipped through the adapter. 
> > > > 
> > > the ftdi will just send data it receives it on the uart side over USB as it fills frames and the rest when some idle period occurred 
> > > 
> > > on the PC side you just read X*Y bytes with a timeout and either you get it or you get a timeout and how many bytes you did get 
> > You aren't following the entire matter.
> I do believe I am ... you want to do ping-pong messaging with a microsecond latencies and that is just not going to happen with a PC unless you use a PCI card and write a custom kernel driver 
> 
> you have to "pipeline" it so you send a big message and get a big reply

There's no indication  of timing issues in the PC.   I would not expect the vendors to tell me the delays are in their product when it is in the PC.  


> >The problem with sending individual messages is, there are delays, latencies, in handling the data from one side to the other in the adapter. In the case of the USB device, I'm let to believe this is because of intentional pausing. They have not indicated the purpose, but in one direction it may be a timeout, so that if no more data is received, the block of data is sent, even if not full (slaves to master). 
> > 
> > In the original approach, both directions would have the delay, because only one message was sent, then the master would wait for the reply. So the master message (command) would see a wait for a "slot" on the polled USB bus, then the reply would see a similar wait. By concatenating many commands from the master into one message, the waits are reduced. But this is only half of the problem. The slave replies must also be sent together to eliminate the many waits on the reply side. Since I don't know for sure the nature of the waits, I can't be sure if the short delays between the slave replies will get around the waits.
> the replies gets buffered in the FTDI, if the slaves all reply before the next slot you get them all 

It's not relevant that all the replies have been sent before the next "slot".  In fact, the message pair takes around 100 uS at 3 Mbps on the serial bus.  That's 10 message pairs in 1 ms.  There are up to 128 end points that need to addressed with a command in each command batch.  So one command to all end points will take 13 ms.  


> is it an issue that it takes a few frames, when it is for all the slave responses and not per slave response ?

I'm not clear what you are asking.  

The original protocol simply is a command and a reply.  The operations are all reading or writing a register, very simple.  Every command is echoed in the reply, plus a response data byte.  Obviously, for any given end point, the response must be returned, before the next command can be sent out.  Sometimes the next command is conditional on the response, such as a polling loop, waiting for some event to be completed, as in a status bit.  But the different end points are all independent.  However, for simplicity, the code will send the same commands to all endpoints.  


> > > > These will show up oddly in a debugger. 
> > > > 
> > > > One vendor suggested I use their product and customize the software myself. It runs under linux. I think I would prefer to build a separate unit, that sits between the serial port and the test fixture bus, separating the commands and waiting for the reply. Actually, I could incorporate this into the test fixture. So the first board receives the data stream from the PC through a separate port, and sends the commands, one a a time, with a bit of space. No need for idle characters to be sent. 
> > > > 
> > > with a MCU it would be very trivial to send/recieve messages with fixed timing, but PC is like a semitruck it can move 
> > > lots of stuff fast but it isn't very good at reacting fast to changes 
> > I think the low level drivers have adequate priority to handle the traffic. At least one vendor has done tests which I believe time the sending on the PC to the receiving on the PC. So even if the PC makes delays, they are factored in. 
> > 
> > Of course, I can't know for sure exactly how they did the tests. One thing that reduces the times is they looped back the data, so one transmission time on the serial bus was not included. But that's not large, so can probably be ignored. 
> > 
> > If the aggregated messages work, that will be the easiest method so far. I've gotten a lot of advice that is from left field, where people didn't take the time to actually read what I had written before replying. Free advice, worth the price. lol 
> > 
> > I really am surprised at the delays in the Ethernet devices. It never occurred to me they would run all the data for 16 serial ports and the Ethernet bus through a single CPU. It's surprising it works well at all. For me, it doesn't. 
> > 
> > About the fast bus not reacting quickly to changes, I've tried to explain to people before, that the Internet isn't just about the "speed" numbers various web sites measure. If you are watching a movie, you need speed. If you are browsing the web, you want low latency as well. No one knows what that means. I try to explain it's the difference between a road having a high speed limit, but being full of traffic lights. Most don't really get it still. 
> >
> watching a movie or browsing the web doesn't need that low latency, you won't notice latency until it gets close to several 10s of milliseconds , a 50Hz monitor is 20ms

I've never had an Internet connection that wasn't at least 10's of ms.  That is a good connection.  It's the 100's of ms that make for slow page loading.  My current connection shows a 13 ms ping to 8.8.8.8, often rising into the 100's of ms.  No impact on streaming videos, but it can be a PITA for page loading.  

Maybe that's how I should describe the analogy.  They all use fast trucks, but the package can sit in a warehouse waiting to be loaded onto the next truck.  It also has to wait on the paperwork at each end of the route.  Some vendors take days to get your order to the loading dock.  Digikey ships the same day with the same carrier!  

-- 

Rick C.

--+-- Get 1,000 miles of free Supercharging
--+-- Tesla referral code - https://ts.la/richard11209

On Sat, 3 Dec 2022 12:57:45 -0800 (PST), Ricky
<gnuarm.deletethisbit@gmail.com> wrote:

>On Saturday, December 3, 2022 at 7:00:04 AM UTC-5, upsid...@downunder.com wrote:
>> On Sat, 3 Dec 2022 00:10:13 -0800 (PST), Ricky 
>> <gnuarm.del...@gmail.com> wrote: 
>> 
>> > 
>> >> IMHO the simplest thing to make the connection would to use an 
>> >> Ethernet pot on a lap/table top, use two 8 port (or four 4 port) 
>> >> Ethernet to RS-42//485 converters. This you would have 16 independent 
>> >> multidrop buses. Put the converters in UDP mode and the programming is 
>> >> easy. Write the whole Tx message into the socket and receive a full 
>> >> serial frame from the UDP socket. 
>> > 
>> >What programming? The serial converters have drivers to make them look like COM ports on the PC. The code to talk to the serial port is already written and working.
>> The problem with those vendor supplied COM port look-a-likes is that 
>> they only support a single line. If you have multiple lines, you will 
>> have to use a process (or at least a thread) for each physical serial 
>> line to get any advantage of using multiple lines. 
>
>I will look into that.  I've not been told I can't open multiple COM ports from a single process. 


Sure you can open multiple serial lines in a single thread, but the
real issue is, can you use them simultaneously during the same
millisecond. Your specifications seem to require multiple transactions
within a millisecond. Doing that from Windows may require that you
split the slaves into multiple serial buses and run the buses in
parallel. 

The simplest vase is to assign a separate thread for each serial line.
The requests from the user mode threads will go through very effective
kernel mode driver handling several requests each millisecond.

If you insist on a single thread using Windows specific functions, you
can use overlapped I/O using ReadFileEx and WiteFileEx (NOT
ReadFile/WriteFile) to run multiple serial lines concurrently. It is
unclear if any adapter COM port look-a-like support
ReadFileEx/WiteFileEx at all or does it properly.

A third alternative with single thread is to use Ehernet/serial
adapters in UDP mode and use socket I/O. The advantage is that it
works on Linux or other Unix with minimal changes.

>
>> Using UDP sockets, you can send a single request to every serial line 
>> in parallel. Then start waiting for responses from these lines, by 
>> issuing a select() call and process each response as it has arrived. 
>> This can all be done un a single thread. I have used this technique to 
>> drive up to 50 serial lines with Eth/serial converters controlling 50 
>> Variable Frequency Drives (VFD) running 50 motors from a single 
>> thread.
>
>How does the UDP socket map to a serial port?  

One windows uses port 4001 for the first serial port, 4002 for second
port and so on. You mist set a unique IP address for each converter
box. To send data to port 8 on a converter at 192.168.1.7 you send the
data string to 192.168.1.7:4008.

>What driver is required from the adapter vendor? 

None. You just need to set the line speed and 2/4Wire RS-4xx mode for
each serial port, set each port into UDP mode and set a unique IP
address for the entire converter box. On the master PC you must just
use socket reads and writes.

On 12/4/2022 12:26 PM, upsidedown@downunder.com wrote:
> None. You just need to set the line speed and 2/4Wire RS-4xx mode for
> each serial port, set each port into UDP mode and set a unique IP
> address for the entire converter box. On the master PC you must just
> use socket reads and writes.

The problem with stateful devices between "here" and "there" is
you have to either ensure (punishable by death) that no one
disconnects/powers-down one of those devices (which would cause it
to lose its current state/configuration); or, build ack/nak into
your protocol so you are sure every external device in your chain
can be verified as "present" and functioning as intended.

[No acks with UDP; TCP considerably more costly; roll-your own may
be even costlier -- esp if no experience designing reliable protocols]

I designed/built a device to allow us to ISP OTPs to speed up
manufacturing.  As it was driven by an '11, my interface had to
be a serial port.  Transfering 128K binaries at 9600 bps is a
considerable timesink -- so, cache the image in the device and
just let the '11 tell you when to write, read, verify, etc.

Easy peasy.

Call from manufacturing some months later that lots of units
were failing final test.  Turned out, the OTPs were blank!

No, not all of them.  Approx one system per day was correct.
The *first*!

Turns out, new hire was "exceedingly prudent" and would power
down my device before disconnecting from the UUT.  Of course,
128K cache was DRAM (FLASH not being available, back then,
and the device wasn't designed to NEED a nonvolatile store!)
so it promptly forgot everything it had been told.

Including start/end addresses, etc.

But, would dutifully report that it had written and verified
every address -- from 0x00000 to 0x00000 on each of the subsequent
invocations!

Had the guy who had written the UI on the '11 included
verification of parameters after each operation (the protocol
that I designed allowed for the host to write AND read the current
settings), he could have alerted the user to the fact that the
start/end addresses had changed ("OhMiGosh!  How??")
and saved me from intervening.

"Do the instructions tell you to power down the device after
each unit?"

"No, but I thought it was good practice not to disconnect things
with power applied."

"Yes, and I designed this device to allow that to happen, SAFELY,
so we didn't have to keep reloading the image after each unit!"

"Oh"

Modify instructions to include:
     "Do not power down device until all units are processed!"

Likewise, serial port implementation had hardware and software
flow control -- instead of just RxTx -- so the host could tell
that there *was* something out there responding to the
data stream -- instead of just a "dangling wire".

On Sunday, December 4, 2022 at 2:26:35 PM UTC-5, upsid...@downunder.com wrote:
> On Sat, 3 Dec 2022 12:57:45 -0800 (PST), Ricky
> <gnuarm.del...@gmail.com> wrote: 
> 
> >On Saturday, December 3, 2022 at 7:00:04 AM UTC-5, upsid...@downunder.com wrote: 
> >> On Sat, 3 Dec 2022 00:10:13 -0800 (PST), Ricky 
> >> <gnuarm.del...@gmail.com> wrote: 
> >> 
> >> > 
> >> >> IMHO the simplest thing to make the connection would to use an 
> >> >> Ethernet pot on a lap/table top, use two 8 port (or four 4 port) 
> >> >> Ethernet to RS-42//485 converters. This you would have 16 independent 
> >> >> multidrop buses. Put the converters in UDP mode and the programming is 
> >> >> easy. Write the whole Tx message into the socket and receive a full 
> >> >> serial frame from the UDP socket. 
> >> > 
> >> >What programming? The serial converters have drivers to make them look like COM ports on the PC. The code to talk to the serial port is already written and working. 
> >> The problem with those vendor supplied COM port look-a-likes is that 
> >> they only support a single line. If you have multiple lines, you will 
> >> have to use a process (or at least a thread) for each physical serial 
> >> line to get any advantage of using multiple lines. 
> > 
> >I will look into that. I've not been told I can't open multiple COM ports from a single process.
> Sure you can open multiple serial lines in a single thread, but the 
> real issue is, can you use them simultaneously during the same 
> millisecond. Your specifications seem to require multiple transactions 
> within a millisecond. Doing that from Windows may require that you 
> split the slaves into multiple serial buses and run the buses in 
> parallel. 
> 
> The simplest vase is to assign a separate thread for each serial line. 
> The requests from the user mode threads will go through very effective 
> kernel mode driver handling several requests each millisecond. 
> 
> If you insist on a single thread using Windows specific functions, you 
> can use overlapped I/O using ReadFileEx and WiteFileEx (NOT 
> ReadFile/WriteFile) to run multiple serial lines concurrently. It is 
> unclear if any adapter COM port look-a-like support 
> ReadFileEx/WiteFileEx at all or does it properly. 

I'm not really getting what you are talking about.  If each serial port has a separate file handle, and I write to one, it will return immediately on copying the data to a new buffer.  I can then write to another serial port using a different file handle and it will return immediately.  Why would the write delay returning to the application?  It certainly does not wait for the transmission of the data.  


> A third alternative with single thread is to use Ehernet/serial 
> adapters in UDP mode and use socket I/O. The advantage is that it 
> works on Linux or other Unix with minimal changes.
> > 
> >> Using UDP sockets, you can send a single request to every serial line 
> >> in parallel. Then start waiting for responses from these lines, by 
> >> issuing a select() call and process each response as it has arrived. 
> >> This can all be done un a single thread. I have used this technique to 
> >> drive up to 50 serial lines with Eth/serial converters controlling 50 
> >> Variable Frequency Drives (VFD) running 50 motors from a single 
> >> thread. 
> > 
> >How does the UDP socket map to a serial port?
> One windows uses port 4001 for the first serial port, 4002 for second 
> port and so on. You mist set a unique IP address for each converter 
> box. To send data to port 8 on a converter at 192.168.1.7 you send the 
> data string to 192.168.1.7:4008.
> >What driver is required from the adapter vendor?
> None. You just need to set the line speed and 2/4Wire RS-4xx mode for 
> each serial port, set each port into UDP mode and set a unique IP 
> address for the entire converter box. On the master PC you must just 
> use socket reads and writes.

I'm not following.  If there is no driver from the comm adapter box vendor, is this UDP port access some sort of universal convention?  Sorry, but I know little about Ethernet.  That is why I'm not eager to dive into this.  I have a lot of other work to do,  and this is taking far too much of my time.  I fully expected the serial port to be the speed limiting factor in the comms, not the issues with USB and Ethernet. 

-- 

Rick C.

--+-+ Get 1,000 miles of free Supercharging
--+-+ Tesla referral code - https://ts.la/richard11209

On Sunday, 4 December 2022 at 21:46:08 UTC, Don Y wrote:
> On 12/4/2022 12:26 PM, upsid...@downunder.com wrote: 
> > None. You just need to set the line speed and 2/4Wire RS-4xx mode for 
> > each serial port, set each port into UDP mode and set a unique IP 
> > address for the entire converter box. On the master PC you must just 
> > use socket reads and writes.
> The problem with stateful devices between "here" and "there" is 
> you have to either ensure (punishable by death) that no one 
> disconnects/powers-down one of those devices (which would cause it 
> to lose its current state/configuration); or, build ack/nak into 
> your protocol so you are sure every external device in your chain 
> can be verified as "present" and functioning as intended. 
> 
> [No acks with UDP; TCP considerably more costly; roll-your own may 
> be even costlier -- esp if no experience designing reliable protocols] 
> 
> I designed/built a device to allow us to ISP OTPs to speed up 
> manufacturing. As it was driven by an '11, my interface had to 
> be a serial port. Transfering 128K binaries at 9600 bps is a 
> considerable timesink -- so, cache the image in the device and 
> just let the '11 tell you when to write, read, verify, etc. 
> 
> Easy peasy. 
> 
> Call from manufacturing some months later that lots of units 
> were failing final test. Turned out, the OTPs were blank! 
> 
> No, not all of them. Approx one system per day was correct. 
> The *first*! 
> 
> Turns out, new hire was "exceedingly prudent" and would power 
> down my device before disconnecting from the UUT. Of course, 
> 128K cache was DRAM (FLASH not being available, back then, 
> and the device wasn't designed to NEED a nonvolatile store!) 
> so it promptly forgot everything it had been told. 
> 
> Including start/end addresses, etc. 
> 
> But, would dutifully report that it had written and verified 
> every address -- from 0x00000 to 0x00000 on each of the subsequent 
> invocations! 
> 
> Had the guy who had written the UI on the '11 included 
> verification of parameters after each operation (the protocol 
> that I designed allowed for the host to write AND read the current 
> settings), he could have alerted the user to the fact that the 
> start/end addresses had changed ("OhMiGosh! How??") 
> and saved me from intervening. 
> 
> "Do the instructions tell you to power down the device after 
> each unit?" 
> 
> "No, but I thought it was good practice not to disconnect things 
> with power applied." 
> 
> "Yes, and I designed this device to allow that to happen, SAFELY, 
> so we didn't have to keep reloading the image after each unit!" 
> 
> "Oh" 
> 
> Modify instructions to include: 
> "Do not power down device until all units are processed!" 
> 
> Likewise, serial port implementation had hardware and software 
> flow control -- instead of just RxTx -- so the host could tell 
> that there *was* something out there responding to the 
> data stream -- instead of just a "dangling wire".

This is a familiar story.  Once upon a time (1975) I was the "new hire".
It was a summer job and one of the tasks was testing a very large pile
of switching power supplies for a communication system in the Stockholm
underground railway.  I was supposed to attach a dummy load to each of the
outputs in turn until the psu was fully loaded, test all the output
voltages and spike levels, then remove the loads one by one. This got very
tedious, so I connected them all at once and then turned on the 48V power.
That was fine until I also disconnected all the loads at once. Unfortunately,
this let out a lot of magic smoke before the input fuse had time to blow.
They had a storage 'scope, so a few power supplies later I found that the
problem was caused by the transient response of the power supply.
It would overshoot on removal of all loads at once.  The linear final
regulators slew rate limited resulting in a triangular output transient just
large enough to trigger a thyristor crowbar.  This caused the power supply to
oscillate at a much higher frequency which exploded the resistor in a
snubber network.
Fortunately, mechanical constraints meant that in normal service it was not
possible to unplug the psu from the rack without first removing the 48V,
so the design didn't need to be changed.
After that I got moved on to more interesting things like testing radar
target simulators.
John