On Tuesday, 6 December 2022 at 18:39:31 UTC, John Larkin wrote:
> On Tue, 6 Dec 2022 08:41:02 -0800 (PST), John Walliker
> <jrwal...@gmail.com> wrote:
>
> >On Monday, 5 December 2022 at 21:06:03 UTC, whit3rd wrote:
> >> On Monday, December 5, 2022 at 7:28:04 AM UTC-8, John Larkin wrote:
> >>
> >> > If we do a product line around raspberry pi, we could piggyback on the
> >> > enormous physical and people culture. I've never seen anything like
> >> > it.
> >> >
> >> > https://www.raspberrypi.org/
> >> >
> >> > We might sponsor 5 or 10 smart poor high school or college kids, steer
> >> > their paths a bit, give them summer projects or jobs, hire a couple of
> >> > the best when they graduate.
> >> >
> >> > Pi has enormous momentum so should be around for a while.
> >> Momentum isn't the correct model for something that can become
> >> nonexistent. The persistent availability of Broadcom SOC chips is
> >> not like the mass of a juggernaut. It is not a physical constant.
> >>
> >> Where is a second source?
> >> Banana pi, anyone?
> >The Pi that JL is using isn't based on Broadcom. Its a really
> >nice ARM M0+ dual core microcontroller, the RP2040 with
> >some excellent i/o capabilities.
> https://en.wikipedia.org/wiki/Raspberry_Pi
>
> Broadcom is involved somehow.
>
> Who manufactures the RP2040? That's not obvious.
> >
> >"Obsolescence statement
> >Raspberry Pi understands the value to customers of long term availability
> >of product and therefore aims to continue supply for as long as practically
> >possible. We expect RP2040 to remain in production until at least January 2041."
> >
> >https://datasheets.raspberrypi.com/rp2040/rp2040-product-brief.pdf
> >https://datasheets.raspberrypi.com/rp2040/rp2040-datasheet.pdf
> >https://www.digikey.co.uk/en/products/filter/microcontrollers/685?s=N4IgTCBcDaIE4AcwAYAsyQF0C%2BQ
> >
> >The board that JL is looking at is the Pico, a small pcb with the RP2040
> >processor, flash memory and a USB interface. There is a version which also has WiFi.
> >Multiple suppliers are making boards which are either clones or in the general
> >spirit of the Pico. JL has already pointed out that he could make his own board
> >if necessary.
> >
> >I am planning to use an RP2040 for switching power supply
> >management among other things. Availability is excellent and its a really
> >nice micro-controller at a good price.
> >
> >John
> The culture and infrastructure around Pi is unprecedented. A million
> kids must be using it. Schools center courses on it.
>
> I was looking at an Analog Devices RF synthesizer chip. The eval kit
> uses a Pi.
I don't think there is any connection between the RP2040 and Broadcom.
The chips are designed by Pi and made by TSMC.
All other Raspberry Pi products do use Broadcom chips. The RP2040
is a very different animal however.
From this article:
https://www.zdnet.com/article/now-you-can-by-giant-reels-of-raspberry-pis-rp2040-chips/
"However, Upton assures people with microcontroller projects that the RP2040
shouldn't suffer the same shortages, with enough wafers to make 20 million chips.
RP2040 is built on a more modern semiconductor process (TSMC 40LP) than
most other microcontrollers. As a result, it makes extremely efficient use of scarce
silicon wafer supply: each die occupies just 2mm2, and each 300mm wafer yields
roughly 21,000 dice. We have sufficient wafer stock on hand to produce 20 million
chips, with more on the way," says Upton.
If you want to build your product on a microcontroller you can actually buy in 2022,
RP2040 is your friend."
John
Reply by John Larkin●December 6, 20222022-12-06
On Tue, 6 Dec 2022 08:41:02 -0800 (PST), John Walliker
<jrwalliker@gmail.com> wrote:
>On Monday, 5 December 2022 at 21:06:03 UTC, whit3rd wrote:
>> On Monday, December 5, 2022 at 7:28:04 AM UTC-8, John Larkin wrote:
>>
>> > If we do a product line around raspberry pi, we could piggyback on the
>> > enormous physical and people culture. I've never seen anything like
>> > it.
>> >
>> > https://www.raspberrypi.org/
>> >
>> > We might sponsor 5 or 10 smart poor high school or college kids, steer
>> > their paths a bit, give them summer projects or jobs, hire a couple of
>> > the best when they graduate.
>> >
>> > Pi has enormous momentum so should be around for a while.
>> Momentum isn't the correct model for something that can become
>> nonexistent. The persistent availability of Broadcom SOC chips is
>> not like the mass of a juggernaut. It is not a physical constant.
>>
>> Where is a second source?
>> Banana pi, anyone?
>The Pi that JL is using isn't based on Broadcom. Its a really
>nice ARM M0+ dual core microcontroller, the RP2040 with
>some excellent i/o capabilities.
>
>"Obsolescence statement
>Raspberry Pi understands the value to customers of long term availability
>of product and therefore aims to continue supply for as long as practically
>possible. We expect RP2040 to remain in production until at least January 2041."
>
>https://datasheets.raspberrypi.com/rp2040/rp2040-product-brief.pdf
>https://datasheets.raspberrypi.com/rp2040/rp2040-datasheet.pdf
>https://www.digikey.co.uk/en/products/filter/microcontrollers/685?s=N4IgTCBcDaIE4AcwAYAsyQF0C%2BQ
>
>The board that JL is looking at is the Pico, a small pcb with the RP2040
>processor, flash memory and a USB interface. There is a version which also has WiFi.
>Multiple suppliers are making boards which are either clones or in the general
>spirit of the Pico. JL has already pointed out that he could make his own board
>if necessary.
>
>I am planning to use an RP2040 for switching power supply
>management among other things. Availability is excellent and its a really
>nice micro-controller at a good price.
>
>John
The culture and infrastructure around Pi is unprecedented. A million
kids must be using it. Schools center courses on it.
I was looking at an Analog Devices RF synthesizer chip. The eval kit
uses a Pi.
Reply by John Walliker●December 6, 20222022-12-06
On Monday, 5 December 2022 at 21:06:03 UTC, whit3rd wrote:
> On Monday, December 5, 2022 at 7:28:04 AM UTC-8, John Larkin wrote:
>
> > If we do a product line around raspberry pi, we could piggyback on the
> > enormous physical and people culture. I've never seen anything like
> > it.
> >
> > https://www.raspberrypi.org/
> >
> > We might sponsor 5 or 10 smart poor high school or college kids, steer
> > their paths a bit, give them summer projects or jobs, hire a couple of
> > the best when they graduate.
> >
> > Pi has enormous momentum so should be around for a while.
> Momentum isn't the correct model for something that can become
> nonexistent. The persistent availability of Broadcom SOC chips is
> not like the mass of a juggernaut. It is not a physical constant.
>
> Where is a second source?
> Banana pi, anyone?
The Pi that JL is using isn't based on Broadcom. Its a really
nice ARM M0+ dual core microcontroller, the RP2040 with
some excellent i/o capabilities.
"Obsolescence statement
Raspberry Pi understands the value to customers of long term availability
of product and therefore aims to continue supply for as long as practically
possible. We expect RP2040 to remain in production until at least January 2041."
https://datasheets.raspberrypi.com/rp2040/rp2040-product-brief.pdfhttps://datasheets.raspberrypi.com/rp2040/rp2040-datasheet.pdfhttps://www.digikey.co.uk/en/products/filter/microcontrollers/685?s=N4IgTCBcDaIE4AcwAYAsyQF0C%2BQ
The board that JL is looking at is the Pico, a small pcb with the RP2040
processor, flash memory and a USB interface. There is a version which also has WiFi.
Multiple suppliers are making boards which are either clones or in the general
spirit of the Pico. JL has already pointed out that he could make his own board
if necessary.
I am planning to use an RP2040 for switching power supply
management among other things. Availability is excellent and its a really
nice micro-controller at a good price.
John
Reply by whit3rd●December 5, 20222022-12-05
On Monday, December 5, 2022 at 7:28:04 AM UTC-8, John Larkin wrote:
> If we do a product line around raspberry pi, we could piggyback on the
> enormous physical and people culture. I've never seen anything like
> it.
>
> https://www.raspberrypi.org/
>
> We might sponsor 5 or 10 smart poor high school or college kids, steer
> their paths a bit, give them summer projects or jobs, hire a couple of
> the best when they graduate.
>
> Pi has enormous momentum so should be around for a while.
Momentum isn't the correct model for something that can become
nonexistent. The persistent availability of Broadcom SOC chips is
not like the mass of a juggernaut. It is not a physical constant.
Where is a second source?
Banana pi, anyone?
Reply by John Larkin●December 5, 20222022-12-05
On Mon, 5 Dec 2022 11:01:34 +0000, Martin Brown
<'''newspam'''@nonad.co.uk> wrote:
>On 02/12/2022 15:22, John Larkin wrote:
>> On Fri, 2 Dec 2022 12:15:56 +0000, Martin Brown
>> <'''newspam'''@nonad.co.uk> wrote:
>>
>>> On 27/11/2022 16:16, John Larkin wrote:
>>>> On Sun, 27 Nov 2022 10:46:47 +0100, Gerhard Hoffmann <dk4xp@arcor.de>
>>>> wrote:
>>>>
>>>>> Am 27.11.22 um 05:34 schrieb John Larkin:
>>>>>> We use the efinix T20 trion FPGA.
>>>>>>
>>>>>> Questions about the config bit streams:
>>>>>>
>>>>>> Are they always the same size, or does it depend on how much logic is
>>>>>> compiled? Would a simple application use less?
>>>>>
>>>>> With Xilinx it would for sure. Never used efinix, but I would
>>>>> consider it broken if it didn't.
>>>>>
>>>>>> Are the streams very compressible?
>>>>>
>>>>> I would simply test example files with zip, zcat and similar.
>>>>> IIRC, there is even a flow-through decompressor.
>>>>>
>>>>> We have done some simple run-length
>>>>>> coding to greatly reduce the storage requirement for other FPGAs.
>>>>>> Configs tend to have long runs of 0's.
>>>>>>
>>>>>> The T20/256 claims to need 5.4 megabits. I'd like to store the fpga
>>>>>> config and application code in a Raspberry Pi Pico, which has 2 MB of
>>>>>> onboard flash. Storing the full config would use about a third of
>>>>>> that, so reducing that would be useful.
>>>>>
>>>>> cheers, Gerhard
>>>>>
>>>>
>>>> I'm at home and don't have access to a compiled bitstream, and this is
>>>> a discussion group.
>>>>
>>>> I'll get a T20 bit stream Monday or Tuesday and see what it looks
>>>> like. If there are many runs of 0's, compression and decompression are
>>>> very simple. Or maybe a typical stream is just shorter than the max.
>>>
>>> Binary looks to have incredibly high redundancy and compressibility.
>>> One of the lowest byte entropy scores I have seen in a long time.
>>
>> My comment was about really random data. An FPGA bit stream certainly
>> has repeated patterns. One might build a N-bit structure, a multiplier
>> or accumulator or filter or DDS, and bit-slice blocks are very likely
>> repeated N times.
>
>I don't think an FPGA bitstream is anything remotely like random data.
>The vast majority of the bytes are zeroes (70%), then bytes with 1 bit
>set ~2% each, 2 bits set <0.7%. It depends how hard you are prepared to
>work. Bytes with more than 3 bits set are comparatively rare.
>
>In your example the bytes 8A, A7, BF, DB, ED all appeared just once and
>the token BE did not occur at all.
>
>In principle for this application you can afford to use insane amounts
>of CPU power to encode if it makes the decoder simpler and faster. My
>instinct is that it is only worth compressing enough to make room for
>whatever code has to fit into the same space.
>
>I recall way back jumping through endless hoops to fit slightly more
>firmware code into 8k ROMs back in the days when 64k was a lot of ram.
>
>> Maybe I can find some college kid who'd like to do a project or thesus
>> to find or code a minimal decomp algorithm for efinix+rasperry pi, in
>> exchange for some pittance.
>
>I used to have a university sandwich student for a year and sometimes a
>student over the long vacation and give them projects that were
>interesting and otherwise wouldn't get done. The occasional one turned
>out to be exceptionally good. The rest did an OK job. It is only worth
>doing if they can finish a project that you don't have the time to do.
>
>Usually something that involves taking a lot of raw data and looking to
>see if there is anything interesting going on.
>>
>> I can imagine some dictionary-based thing where a dictionary entry is
>> its own first occurrence in the bit file. The decompressor is
>> basically scissors and a pot of glue.
>
>Judging by the way it looks to my correlator I would expect LHA type
>algorithms to do rather well on it. There is an inordinate amount of
>block duplication. A few simple subs will easily get you under 250k.
>
>>> There appear to be strong correlations of identical blocks at strides of
>>> 9, 12, 24, 36 as well as huge runs of nul bytes. The odd one of 0a.
>>>
>>> Also a quick eyeball reveals walking ones 80,40,20,10,08,04,02,01,00
>>> at around 107227 (stride 9).
>>>
>>> There is an incredibly long run of 15372 nul bytes at offset 143811
>>>
>>> RLE the nul bytes should get you most of the way there and maybe some
>>> code to RLE the most obvious repeated sequences if you need a bit more.
>>
>> I was thinking of just compressing runs of 0's, but there could be a
>> few other smallish patterns that might not be horrible to stash in the
>> decompressor dictionary. That presents the question, are there
>> patterns that are common to *all* T20 bit streams?
>>
>> I need a low-paid lackey.
>
>What stops you from having one?
>But you will get more use out of one that is paid the going rate.
Just kidding. We pay very well.
If we do a product line around raspberry pi, we could piggyback on the
enormous physical and people culture. I've never seen anything like
it.
https://www.raspberrypi.org/
We might sponsor 5 or 10 smart poor high school or college kids, steer
their paths a bit, give them summer projects or jobs, hire a couple of
the best when they graduate.
Pi has enormous momentum so should be around for a while.
Reply by Martin Brown●December 5, 20222022-12-05
On 02/12/2022 15:22, John Larkin wrote:
> On Fri, 2 Dec 2022 12:15:56 +0000, Martin Brown
> <'''newspam'''@nonad.co.uk> wrote:
>
>> On 27/11/2022 16:16, John Larkin wrote:
>>> On Sun, 27 Nov 2022 10:46:47 +0100, Gerhard Hoffmann <dk4xp@arcor.de>
>>> wrote:
>>>
>>>> Am 27.11.22 um 05:34 schrieb John Larkin:
>>>>> We use the efinix T20 trion FPGA.
>>>>>
>>>>> Questions about the config bit streams:
>>>>>
>>>>> Are they always the same size, or does it depend on how much logic is
>>>>> compiled? Would a simple application use less?
>>>>
>>>> With Xilinx it would for sure. Never used efinix, but I would
>>>> consider it broken if it didn't.
>>>>
>>>>> Are the streams very compressible?
>>>>
>>>> I would simply test example files with zip, zcat and similar.
>>>> IIRC, there is even a flow-through decompressor.
>>>>
>>>> We have done some simple run-length
>>>>> coding to greatly reduce the storage requirement for other FPGAs.
>>>>> Configs tend to have long runs of 0's.
>>>>>
>>>>> The T20/256 claims to need 5.4 megabits. I'd like to store the fpga
>>>>> config and application code in a Raspberry Pi Pico, which has 2 MB of
>>>>> onboard flash. Storing the full config would use about a third of
>>>>> that, so reducing that would be useful.
>>>>
>>>> cheers, Gerhard
>>>>
>>>
>>> I'm at home and don't have access to a compiled bitstream, and this is
>>> a discussion group.
>>>
>>> I'll get a T20 bit stream Monday or Tuesday and see what it looks
>>> like. If there are many runs of 0's, compression and decompression are
>>> very simple. Or maybe a typical stream is just shorter than the max.
>>
>> Binary looks to have incredibly high redundancy and compressibility.
>> One of the lowest byte entropy scores I have seen in a long time.
>
> My comment was about really random data. An FPGA bit stream certainly
> has repeated patterns. One might build a N-bit structure, a multiplier
> or accumulator or filter or DDS, and bit-slice blocks are very likely
> repeated N times.
I don't think an FPGA bitstream is anything remotely like random data.
The vast majority of the bytes are zeroes (70%), then bytes with 1 bit
set ~2% each, 2 bits set <0.7%. It depends how hard you are prepared to
work. Bytes with more than 3 bits set are comparatively rare.
In your example the bytes 8A, A7, BF, DB, ED all appeared just once and
the token BE did not occur at all.
In principle for this application you can afford to use insane amounts
of CPU power to encode if it makes the decoder simpler and faster. My
instinct is that it is only worth compressing enough to make room for
whatever code has to fit into the same space.
I recall way back jumping through endless hoops to fit slightly more
firmware code into 8k ROMs back in the days when 64k was a lot of ram.
> Maybe I can find some college kid who'd like to do a project or thesus
> to find or code a minimal decomp algorithm for efinix+rasperry pi, in
> exchange for some pittance.
I used to have a university sandwich student for a year and sometimes a
student over the long vacation and give them projects that were
interesting and otherwise wouldn't get done. The occasional one turned
out to be exceptionally good. The rest did an OK job. It is only worth
doing if they can finish a project that you don't have the time to do.
Usually something that involves taking a lot of raw data and looking to
see if there is anything interesting going on.
>
> I can imagine some dictionary-based thing where a dictionary entry is
> its own first occurrence in the bit file. The decompressor is
> basically scissors and a pot of glue.
Judging by the way it looks to my correlator I would expect LHA type
algorithms to do rather well on it. There is an inordinate amount of
block duplication. A few simple subs will easily get you under 250k.
>> There appear to be strong correlations of identical blocks at strides of
>> 9, 12, 24, 36 as well as huge runs of nul bytes. The odd one of 0a.
>>
>> Also a quick eyeball reveals walking ones 80,40,20,10,08,04,02,01,00
>> at around 107227 (stride 9).
>>
>> There is an incredibly long run of 15372 nul bytes at offset 143811
>>
>> RLE the nul bytes should get you most of the way there and maybe some
>> code to RLE the most obvious repeated sequences if you need a bit more.
>
> I was thinking of just compressing runs of 0's, but there could be a
> few other smallish patterns that might not be horrible to stash in the
> decompressor dictionary. That presents the question, are there
> patterns that are common to *all* T20 bit streams?
>
> I need a low-paid lackey.
What stops you from having one?
But you will get more use out of one that is paid the going rate.
--
Regards,
Martin Brown
Reply by wmartin●December 2, 20222022-12-02
On 12/1/22 23:57, Lasse Langwadt Christensen wrote:
> fredag den 2. december 2022 kl. 06.23.52 UTC+1 skrev John Larkin:
>> On Fri, 2 Dec 2022 13:21:18 +1100, Clifford Heath <no_...@please.net>
>> wrote:
>>> On 2/12/22 09:06, Ricky wrote:
>>>> On Thursday, December 1, 2022 at 4:51:08 PM UTC-5, Clifford Heath wrote:
>>>>> On 2/12/22 08:12, John Larkin wrote:
>>>>>> On Sat, 26 Nov 2022 20:34:23 -0800, John Larkin
>>>>>> <jla...@highlandSNIPMEtechnology.com> wrote:
>>>>>>
>>>>>>> We use the efinix T20 trion FPGA.
>>>>>>>
>>>>>>> Questions about the config bit streams:
>>>>>>>
>>>>>>> Are they always the same size, or does it depend on how much logic is
>>>>>>> compiled? Would a simple application use less?
>>>>>>>
>>>>>>> Are the streams very compressible? We have done some simple run-length
>>>>>>> coding to greatly reduce the storage requirement for other FPGAs.
>>>>>>> Configs tend to have long runs of 0's.
>>>>>>>
>>>>>>> The T20/256 claims to need 5.4 megabits. I'd like to store the fpga
>>>>>>> config and application code in a Raspberry Pi Pico, which has 2 MB of
>>>>>>> onboard flash. Storing the full config would use about a third of
>>>>>>> that, so reducing that would be useful.
>>>>>>
>>>>>> Here's a T20 bit stream. The length seems to be constant vs functions
>>>>>> coded, but there are enough runs of all 0's that it's probably worth
>>>>>> compressing.
>>>>>>
>>>>>> https://www.dropbox.com/s/vm247lntp78jm20/Efinix_T20_bitstream.hex?dl=0
>>>>>>
>>>>>> The actual config file will be binary, not hex of course.
>>>>> Gzip compresses your 2.0MB down to 105kB. The decompressor isn't tiny,
>>>>> but it's fairly small. The lz4 decompressor is tiny and still gets to
>>>>> 221kB. Possibly less if you RLE first. bz2 gets it to 76kB, and xz or
>>>>> lzma to 72kB.
>>>>>
>>>>> Compression is one area where it's best to rely on work done by people
>>>>> who understand the theory. Some of these algorithms have a tiny
>>>>> decompressor, the magic is in the compressor.
>>>>
>>>> Not really.
>>>
>>> Not really what? Spouting words without meaning again, you are such an
>>> adversarial dope.
>>>
>>> I was talking about the relative code complexity of a compressor
>>> compared to its matching decompressor. A decompressor can be tiny, which
>>> is a quality JL seemed to be concerned with.
>> Yes, we want a very small decompressor.
>>
>> I'm thinking of designing some small boxes with a Raspberry Pi Pico as
>> the computer. Here's the rough idea:
>>
>> https://www.dropbox.com/s/dq7lq9bzcrtf285/PiPico_in_Tbox.jpg?raw=1
>>
>> The Pi has only 2 MB of flash and 256KB of sram. An uncompressed T20
>> binary (not hex) bit stream would use about 1/3 of the flash.
>
> but what would you need much more flash for ?
>
> anyway, https://www.waveshare.com/rp2040-plus.htm
>
>
>
I hope Waveshare did a better job with the board layout than they did
with the pictures on the web site. Several of the pin allocations seem
to disagree with the board silkscreen...!
Reply by Joe Gwinn●December 2, 20222022-12-02
On Fri, 02 Dec 2022 07:22:38 -0800, John Larkin
<jlarkin@highlandSNIPMEtechnology.com> wrote:
>On Fri, 2 Dec 2022 12:15:56 +0000, Martin Brown
><'''newspam'''@nonad.co.uk> wrote:
>
>>On 27/11/2022 16:16, John Larkin wrote:
>>> On Sun, 27 Nov 2022 10:46:47 +0100, Gerhard Hoffmann <dk4xp@arcor.de>
>>> wrote:
>>>
>>>> Am 27.11.22 um 05:34 schrieb John Larkin:
>>>>> We use the efinix T20 trion FPGA.
>>>>>
>>>>> Questions about the config bit streams:
>>>>>
>>>>> Are they always the same size, or does it depend on how much logic is
>>>>> compiled? Would a simple application use less?
>>>>
>>>> With Xilinx it would for sure. Never used efinix, but I would
>>>> consider it broken if it didn't.
>>>>
>>>>> Are the streams very compressible?
>>>>
>>>> I would simply test example files with zip, zcat and similar.
>>>> IIRC, there is even a flow-through decompressor.
>>>>
>>>> We have done some simple run-length
>>>>> coding to greatly reduce the storage requirement for other FPGAs.
>>>>> Configs tend to have long runs of 0's.
>>>>>
>>>>> The T20/256 claims to need 5.4 megabits. I'd like to store the fpga
>>>>> config and application code in a Raspberry Pi Pico, which has 2 MB of
>>>>> onboard flash. Storing the full config would use about a third of
>>>>> that, so reducing that would be useful.
>>>>
>>>> cheers, Gerhard
>>>>
>>>
>>> I'm at home and don't have access to a compiled bitstream, and this is
>>> a discussion group.
>>>
>>> I'll get a T20 bit stream Monday or Tuesday and see what it looks
>>> like. If there are many runs of 0's, compression and decompression are
>>> very simple. Or maybe a typical stream is just shorter than the max.
>>
>>Binary looks to have incredibly high redundancy and compressibility.
>>One of the lowest byte entropy scores I have seen in a long time.
>
>
>My comment was about really random data. An FPGA bit stream certainly
>has repeated patterns. One might build a N-bit structure, a multiplier
>or accumulator or filter or DDS, and bit-slice blocks are very likely
>repeated N times.
>
>Maybe I can find some college kid who'd like to do a project or thesus
>to find or code a minimal decomp algorithm for efinix+rasperry pi, in
>exchange for some pittance.
>
>I can imagine some dictionary-based thing where a dictionary entry is
>its own first occurrence in the bit file. The decompressor is
>basically scissors and a pot of glue.
>
>
>>
>>There appear to be strong correlations of identical blocks at strides of
>>9, 12, 24, 36 as well as huge runs of nul bytes. The odd one of 0a.
>>
>>Also a quick eyeball reveals walking ones 80,40,20,10,08,04,02,01,00
>>at around 107227 (stride 9).
>>
>>There is an incredibly long run of 15372 nul bytes at offset 143811
>>
>>RLE the nul bytes should get you most of the way there and maybe some
>>code to RLE the most obvious repeated sequences if you need a bit more.
>
>I was thinking of just compressing runs of 0's, but there could be a
>few other smallish patterns that might not be horrible to stash in the
>decompressor dictionary. That presents the question, are there
>patterns that are common to *all* T20 bit streams?
>
>I need a low-paid lackey.
On Friday, December 2, 2022 at 10:22:53 AM UTC-5, John Larkin wrote:
> On Fri, 2 Dec 2022 12:15:56 +0000, Martin Brown
> <'''newspam'''@nonad.co.uk> wrote:
>
> >On 27/11/2022 16:16, John Larkin wrote:
> >> On Sun, 27 Nov 2022 10:46:47 +0100, Gerhard Hoffmann <dk...@arcor.de>
> >> wrote:
> >>
> >>> Am 27.11.22 um 05:34 schrieb John Larkin:
> >>>> We use the efinix T20 trion FPGA.
> >>>>
> >>>> Questions about the config bit streams:
> >>>>
> >>>> Are they always the same size, or does it depend on how much logic is
> >>>> compiled? Would a simple application use less?
> >>>
> >>> With Xilinx it would for sure. Never used efinix, but I would
> >>> consider it broken if it didn't.
> >>>
> >>>> Are the streams very compressible?
> >>>
> >>> I would simply test example files with zip, zcat and similar.
> >>> IIRC, there is even a flow-through decompressor.
> >>>
> >>> We have done some simple run-length
> >>>> coding to greatly reduce the storage requirement for other FPGAs.
> >>>> Configs tend to have long runs of 0's.
> >>>>
> >>>> The T20/256 claims to need 5.4 megabits. I'd like to store the fpga
> >>>> config and application code in a Raspberry Pi Pico, which has 2 MB of
> >>>> onboard flash. Storing the full config would use about a third of
> >>>> that, so reducing that would be useful.
> >>>
> >>> cheers, Gerhard
> >>>
> >>
> >> I'm at home and don't have access to a compiled bitstream, and this is
> >> a discussion group.
> >>
> >> I'll get a T20 bit stream Monday or Tuesday and see what it looks
> >> like. If there are many runs of 0's, compression and decompression are
> >> very simple. Or maybe a typical stream is just shorter than the max.
> >
> >Binary looks to have incredibly high redundancy and compressibility.
> >One of the lowest byte entropy scores I have seen in a long time.
> My comment was about really random data. An FPGA bit stream certainly
> has repeated patterns. One might build a N-bit structure, a multiplier
> or accumulator or filter or DDS, and bit-slice blocks are very likely
> repeated N times.
You are thinking only of the logic. Logic is a small fraction of the bit stream. It is the routing that takes up the bulk of it. That is much less likely to fit repeated patterns because it contains multiple levels of routing in the same region. But I don't know the structure details. I haven't reverse engineered any FPGAs... yet.
> Maybe I can find some college kid who'd like to do a project or thesus
> to find or code a minimal decomp algorithm for efinix+rasperry pi, in
> exchange for some pittance.
>
> I can imagine some dictionary-based thing where a dictionary entry is
> its own first occurrence in the bit file. The decompressor is
> basically scissors and a pot of glue.
I think some thoughtful analysis of the matter, or even simply researching the issue, would be much more productive. In fact, this would be a great project for a student, part time, while at school. Find out what the other 512,976 FPGA designers are using for compression. Oh, that's what Larkin is doing by posting here. And he doesn't have to pay anyone!
Talk about low paid lackeys!
> >There appear to be strong correlations of identical blocks at strides of
> >9, 12, 24, 36 as well as huge runs of nul bytes. The odd one of 0a.
> >
> >Also a quick eyeball reveals walking ones 80,40,20,10,08,04,02,01,00
> >at around 107227 (stride 9).
> >
> >There is an incredibly long run of 15372 nul bytes at offset 143811
> >
> >RLE the nul bytes should get you most of the way there and maybe some
> >code to RLE the most obvious repeated sequences if you need a bit more.
> I was thinking of just compressing runs of 0's, but there could be a
> few other smallish patterns that might not be horrible to stash in the
> decompressor dictionary. That presents the question, are there
> patterns that are common to *all* T20 bit streams?
>
> I need a low-paid lackey.
Or, you can ask in appropriate forums where you can find people who actually work with Efinix. Or maybe call Efinix and ask them about compression. Just a thought, but maybe they know a little bit about it?
--
Rick C.
--- Get 1,000 miles of free Supercharging
--- Tesla referral code - https://ts.la/richard11209
Reply by Ricky●December 2, 20222022-12-02
On Friday, December 2, 2022 at 8:32:38 AM UTC-5, Martin Brown wrote:
> On 02/12/2022 00:07, Ricky wrote:
> > On Thursday, December 1, 2022 at 6:06:03 PM UTC-5, Martin Brown
> > wrote:
> >> On 01/12/2022 21:12, John Larkin wrote:
> >>> On Sat, 26 Nov 2022 20:34:23 -0800, John Larkin
> >>> <jla...@highlandSNIPMEtechnology.com> wrote:
> >>>
> >>>> We use the efinix T20 trion FPGA.
> >>>>
> >>>> Questions about the config bit streams:
> >>>>
> >>>> Are they always the same size, or does it depend on how much
> >>>> logic is compiled? Would a simple application use less?
> Possibly. There are a hell of a lot of nulls in the binary ~70%
> Some very long sequences of >15k nulls too. All the rest are a few
> hundred or less. My program sees 17 blocks of 15372 nulls in the file.
> (it is expecting a damaged JPEG)
>
> 15372 = 2^2.3^2.7.61
>
> Which seems to me a very odd random constant length for a block!
There is structure at every level. In a less populated design there may be blocks that are totally unused. Are "nulls" bits, or bytes? The compression may work better at the bit level, but if you are seeing such large structures, maybe not. I wonder if you would see more of these repeating structures at the bit level? If they are not aligned to the bytes, they would not appear in a byte level analysis in the same way.
> >> Back of the envelope RLE might get you a ~20x decrease in size.
I would speculate that a 20x compression would not be useful, because of the differences in final size based on the utilization of the part, or even just random factors, such as layout of the same exact design. Move logic around and it can change the empty blocks.
> >> The right compressor and it could be made a lot smaller. If you put
> >> up the binary I'll scan that for byte entropy too.
> >
> > I've never heard of storing the bit stream in an ASCII file. FPGA
> > bit streams are binary data. But maybe I'm just not remembering. It
> > has been a while since I mucked with it at that level.
> I suspect a bit like with EPROM programmers there are development tools
> around which expect a .HEX file. The binary would be much more
> meaningful for working out a compression strategy. FPGA isn't my thing.
Yes, I expect that is correct.
> > I had a design compiled for the 3.3V core voltage version of a chip.
> > There was also a 1.2V core voltage version, which was the same chip,
> > with the LDOs turned into bypass. Seems they use a bond wire to flip
> > a bit in a status register the JTAG reads to distinguish the two.
> > But the JTAG software checks, and you need a file that matches the ID
> > value. I had to find this ID and then recompute the checksum (maybe
> > a simple 8 bit add, rather than a CRC). I think that was an ascii
> > file now that I think of it. I guess they use ASCII to make it
> > easier to see what's what if you have to view it.
> It is more easily readable by a human I suppose.
> >
> > But the underling data is the binary equivalent to the ASCII, so the
> > 3:1 gain of turning the ASCII data into binary is not really
> > relevant. That's more a matter of discarding the pretty printing
> > formatting. In fact, I'm pretty sure the Xilinx bit streams I've
> > seen are binary. There was no translation in sending them to the
> > chips. I expect this file is a .hex simply for purposes of sharing.
> Very probably.
> > The sparsity of non-zero data in the file gives you an idea of the
> > amount of unused resources in FPGAs. That's why they need the latest
> > fab processes to be economical. They have much more silicon area
> > than virtually any other device for the amount of resources actually
> > used.
> I'm curious about the obvious walking ones patterns in it.
It may be a repeated element with a size that is not a multiple of 8.
> The nulls I can account for as unused parts of the functionality.
> The length of them seems peculiar though (I expected 2^N).
I have no idea why. The FPGA has elements of logic and routing. Within those structures are other structures. Few of them will even be a multiple of 8 in length, much less a power of 2.
I recall using gear that looked for repetitions in data by using a waterfall display. Framing bits would stand out as columns of 1s, 0s or even checkerboard patterns. The device I sell uses a three bit framing pattern, if I recall correctly. Because it doesn't vary, it would stand out like a sore thumb once you guess the length of the message right.
In configuration files, I don't think they use framing bits, but the contents will be more obvious when you find the right size for the particular element. Trouble is, there are many, many, different elements.
Not Efinix yet, but people have reverse engineered FPGA configuration files. This is how we have open source tools. When you have small devices, this is easier to do.
--
Rick C.
++ Get 1,000 miles of free Supercharging
++ Tesla referral code - https://ts.la/richard11209