Burn-in strategy

We're making devices that are typically built from 3+ "modules"
(form factor is highly constrained so board space is similarly).

Presently looking at the cost-benefit assessment of burning-in
the modules and/or assembled devices.

*Modules* will be made offshore so any test/burnin has to be part
of the quoted manufacturing cost.

Modules will be "post processed" domestically to install final
firmware, S/Ns, private keys and watermarks.  This allows for
all of that information to be tracked, here (instead of relying
on an offshore vendor who may decide to copy IP).

Aside from installing the above, the only real "mechanical"
modifications are connecting modules together and packaging.

So, a "final test" can just verify proper operation of
the *device* (instead of its constituent modules).

If modules are burned-in and tested prior to acceptance, domesticly,
the number of failures after assembly should be minimal.  (Yet,
you'd still want to verify proper operation as the cost to repair
or replace far exceeds the price of the devices)

OTOH, an assembly problem that could manifest during burn-in
would want some assurances that it wouldn't sneek past final
inspection in the absence of a post-assembly burn-in phase.

At the very least, this sort of question would apply to anyone who
installs firmware after offshore manufacturing.  Or, assembles
subsystems sourced offshore.

So, what is the best practices guidance?

On Fri, 2 Dec 2022 10:53:51 -0700, Don Y <blockedofcourse@foo.invalid>
wrote:

>We're making devices that are typically built from 3+ "modules"
>(form factor is highly constrained so board space is similarly).
>
>Presently looking at the cost-benefit assessment of burning-in
>the modules and/or assembled devices.
>
>*Modules* will be made offshore so any test/burnin has to be part
>of the quoted manufacturing cost.
>
>Modules will be "post processed" domestically to install final
>firmware, S/Ns, private keys and watermarks.  This allows for
>all of that information to be tracked, here (instead of relying
>on an offshore vendor who may decide to copy IP).
>
>Aside from installing the above, the only real "mechanical"
>modifications are connecting modules together and packaging.
>
>So, a "final test" can just verify proper operation of
>the *device* (instead of its constituent modules).
>
>If modules are burned-in and tested prior to acceptance, domesticly,
>the number of failures after assembly should be minimal.  (Yet,
>you'd still want to verify proper operation as the cost to repair
>or replace far exceeds the price of the devices)
>
>OTOH, an assembly problem that could manifest during burn-in
>would want some assurances that it wouldn't sneek past final
>inspection in the absence of a post-assembly burn-in phase.
>
>At the very least, this sort of question would apply to anyone who
>installs firmware after offshore manufacturing.  Or, assembles
>subsystems sourced offshore.
>
>So, what is the best practices guidance?

We were doing overnight burnin+test of our products, after automated
test and cal, but the failure rate was zero. Useful burnin might take
weeks and temperature cycling or something expensive like that.

Temperature cycling is probably the biggest stressor of parts and
solder joints and design margins. Shock+vibration next. Just benign
burnin doesn't seem to do much.

On Friday, December 2, 2022 at 12:54:08 PM UTC-5, Don Y wrote:
> We're making devices that are typically built from 3+ "modules" 
> (form factor is highly constrained so board space is similarly). 
> 
> Presently looking at the cost-benefit assessment of burning-in 
> the modules and/or assembled devices. 
> 
> *Modules* will be made offshore so any test/burnin has to be part 
> of the quoted manufacturing cost. 
> 
> Modules will be "post processed" domestically to install final 
> firmware, S/Ns, private keys and watermarks. This allows for 
> all of that information to be tracked, here (instead of relying 
> on an offshore vendor who may decide to copy IP). 

Sorry, but how can you do burn in offshore, if the modules have no firmware?  


> Aside from installing the above, the only real "mechanical" 
> modifications are connecting modules together and packaging. 
> 
> So, a "final test" can just verify proper operation of 
> the *device* (instead of its constituent modules). 
> 
> If modules are burned-in and tested prior to acceptance, domesticly, 
> the number of failures after assembly should be minimal. (Yet, 
> you'd still want to verify proper operation as the cost to repair 
> or replace far exceeds the price of the devices) 

So the modules will be *built* offshore, but tested and burned in domestically?  I am surprised at the number of faults we have in production, which are claimed to be bad parts!  That's hard for me to believe.  The parts are tested pathologically and handled like the crown jewels in regards to anything that would harm them.  But like Larkin, we see almost no failures in overnight "burn in" which is at room temperature.  I'm ok with that. 

I am making test more automated.  Designing a fixture with 16 boards that each hold 8 UUTs.  This will fit in a 6U chassis and run over 1,000 repetitions of the test overnight.  If there are no failures, that would provide high confidence the units are working. 


> OTOH, an assembly problem that could manifest during burn-in 
> would want some assurances that it wouldn't sneek past final 
> inspection in the absence of a post-assembly burn-in phase. 

I don't know your product, but it's hard to imagine any assembly error that would not show up in test, but would show up after burn in.  

As a data point, I was once told, in a quality meeting, that brand Y TVs are first turned on as a unit, by the customer.  They test every component enough, that they would catch very, very few failures in the final unit.  That said, around the same time, a friend bought a TV that you could turn on with the remote, but could only turn off by pulling the plug!  LOL!!!  I'm willing to bet if they had a final test that turned on the TV by the remote, they would just cut power, rather than turn it off by the remote.  


> At the very least, this sort of question would apply to anyone who 
> installs firmware after offshore manufacturing. Or, assembles 
> subsystems sourced offshore. 
> 
> So, what is the best practices guidance?

I don't think this group will be able to offer much in the way of "best practices", just personal experience. 

The only reason I'm even doing burn in, is to provide a level of confidence in my customer... and a means of verifying the CM is doing a good job.  I use local people, because my product is not yet in a stage where it could be fabbed and tested without my involvement, up front, at least.  It's a long drive to Taiwan. 

-- 

Rick C.

- Get 1,000 miles of free Supercharging
- Tesla referral code - https://ts.la/richard11209

On Fri, 02 Dec 2022 10:07:54 -0800, John Larkin
<jlarkin@highlandSNIPMEtechnology.com> wrote:

>On Fri, 2 Dec 2022 10:53:51 -0700, Don Y <blockedofcourse@foo.invalid>
>wrote:
>
>>We're making devices that are typically built from 3+ "modules"
>>(form factor is highly constrained so board space is similarly).
>>
>>Presently looking at the cost-benefit assessment of burning-in
>>the modules and/or assembled devices.
>>
>>*Modules* will be made offshore so any test/burnin has to be part
>>of the quoted manufacturing cost.
>>
>>Modules will be "post processed" domestically to install final
>>firmware, S/Ns, private keys and watermarks.  This allows for
>>all of that information to be tracked, here (instead of relying
>>on an offshore vendor who may decide to copy IP).
>>
>>Aside from installing the above, the only real "mechanical"
>>modifications are connecting modules together and packaging.
>>
>>So, a "final test" can just verify proper operation of
>>the *device* (instead of its constituent modules).
>>
>>If modules are burned-in and tested prior to acceptance, domesticly,
>>the number of failures after assembly should be minimal.  (Yet,
>>you'd still want to verify proper operation as the cost to repair
>>or replace far exceeds the price of the devices)
>>
>>OTOH, an assembly problem that could manifest during burn-in
>>would want some assurances that it wouldn't sneek past final
>>inspection in the absence of a post-assembly burn-in phase.
>>
>>At the very least, this sort of question would apply to anyone who
>>installs firmware after offshore manufacturing.  Or, assembles
>>subsystems sourced offshore.
>>
>>So, what is the best practices guidance?
>
>We were doing overnight burnin+test of our products, after automated
>test and cal, but the failure rate was zero. Useful burnin might take
>weeks and temperature cycling or something expensive like that.
>
>Temperature cycling is probably the biggest stressor of parts and
>solder joints and design margins. Shock+vibration next. Just benign
>burnin doesn't seem to do much.

Yes.  See Accelerated Life Cycle Testing:

.<https://en.wikipedia.org/wiki/Accelerated_life_testing#:~:text=Accelerated%20life%20testing%20is%20the,a%20short%20amount%20of%20time.>

Joe Gwinn

On Friday, December 2, 2022 at 1:08:08 PM UTC-5, John Larkin wrote:
> On Fri, 2 Dec 2022 10:53:51 -0700, Don Y <blocked...@foo.invalid> 
> wrote:
> >We're making devices that are typically built from 3+ "modules" 
> >(form factor is highly constrained so board space is similarly). 
> > 
> >Presently looking at the cost-benefit assessment of burning-in 
> >the modules and/or assembled devices. 
> > 
> >*Modules* will be made offshore so any test/burnin has to be part 
> >of the quoted manufacturing cost. 
> > 
> >Modules will be "post processed" domestically to install final 
> >firmware, S/Ns, private keys and watermarks. This allows for 
> >all of that information to be tracked, here (instead of relying 
> >on an offshore vendor who may decide to copy IP). 
> > 
> >Aside from installing the above, the only real "mechanical" 
> >modifications are connecting modules together and packaging. 
> > 
> >So, a "final test" can just verify proper operation of 
> >the *device* (instead of its constituent modules). 
> > 
> >If modules are burned-in and tested prior to acceptance, domesticly, 
> >the number of failures after assembly should be minimal. (Yet, 
> >you'd still want to verify proper operation as the cost to repair 
> >or replace far exceeds the price of the devices) 
> > 
> >OTOH, an assembly problem that could manifest during burn-in 
> >would want some assurances that it wouldn't sneek past final 
> >inspection in the absence of a post-assembly burn-in phase. 
> > 
> >At the very least, this sort of question would apply to anyone who 
> >installs firmware after offshore manufacturing. Or, assembles 
> >subsystems sourced offshore. 
> > 
> >So, what is the best practices guidance?
> We were doing overnight burnin+test of our products, after automated 
> test and cal, but the failure rate was zero. Useful burnin might take 
> weeks and temperature cycling or something expensive like that. 
> 
> Temperature cycling is probably the biggest stressor of parts and 
> solder joints and design margins. Shock+vibration next. Just benign 
> burnin doesn't seem to do much.

Infant Mortality--The Lesser Known Reliability Issue
https://ieeexplore.ieee.org/document/4274831

On Fri, 2 Dec 2022 15:09:26 -0800 (PST), Fred Bloggs
<bloggs.fredbloggs.fred@gmail.com> wrote:

>On Friday, December 2, 2022 at 1:08:08 PM UTC-5, John Larkin wrote:
>> On Fri, 2 Dec 2022 10:53:51 -0700, Don Y <blocked...@foo.invalid> 
>> wrote:
>> >We're making devices that are typically built from 3+ "modules" 
>> >(form factor is highly constrained so board space is similarly). 
>> > 
>> >Presently looking at the cost-benefit assessment of burning-in 
>> >the modules and/or assembled devices. 
>> > 
>> >*Modules* will be made offshore so any test/burnin has to be part 
>> >of the quoted manufacturing cost. 
>> > 
>> >Modules will be "post processed" domestically to install final 
>> >firmware, S/Ns, private keys and watermarks. This allows for 
>> >all of that information to be tracked, here (instead of relying 
>> >on an offshore vendor who may decide to copy IP). 
>> > 
>> >Aside from installing the above, the only real "mechanical" 
>> >modifications are connecting modules together and packaging. 
>> > 
>> >So, a "final test" can just verify proper operation of 
>> >the *device* (instead of its constituent modules). 
>> > 
>> >If modules are burned-in and tested prior to acceptance, domesticly, 
>> >the number of failures after assembly should be minimal. (Yet, 
>> >you'd still want to verify proper operation as the cost to repair 
>> >or replace far exceeds the price of the devices) 
>> > 
>> >OTOH, an assembly problem that could manifest during burn-in 
>> >would want some assurances that it wouldn't sneek past final 
>> >inspection in the absence of a post-assembly burn-in phase. 
>> > 
>> >At the very least, this sort of question would apply to anyone who 
>> >installs firmware after offshore manufacturing. Or, assembles 
>> >subsystems sourced offshore. 
>> > 
>> >So, what is the best practices guidance?
>> We were doing overnight burnin+test of our products, after automated 
>> test and cal, but the failure rate was zero. Useful burnin might take 
>> weeks and temperature cycling or something expensive like that. 
>> 
>> Temperature cycling is probably the biggest stressor of parts and 
>> solder joints and design margins. Shock+vibration next. Just benign 
>> burnin doesn't seem to do much.
>
>Infant Mortality--The Lesser Known Reliability Issue
>https://ieeexplore.ieee.org/document/4274831

Paywalled. What's a reasonable burn time to catch infant mortality? If
it's months, it wouldn't be practical for commercial gear.

On Friday, December 2, 2022 at 5:40:38 PM UTC-5, Joe Gwinn wrote:
> On Fri, 02 Dec 2022 10:07:54 -0800, John Larkin 
> <jla...@highlandSNIPMEtechnology.com> wrote: 
> 
> >On Fri, 2 Dec 2022 10:53:51 -0700, Don Y <blocked...@foo.invalid> 
> >wrote: 
> > 
> >>We're making devices that are typically built from 3+ "modules" 
> >>(form factor is highly constrained so board space is similarly). 
> >> 
> >>Presently looking at the cost-benefit assessment of burning-in 
> >>the modules and/or assembled devices. 
> >> 
> >>*Modules* will be made offshore so any test/burnin has to be part 
> >>of the quoted manufacturing cost. 
> >> 
> >>Modules will be "post processed" domestically to install final 
> >>firmware, S/Ns, private keys and watermarks. This allows for 
> >>all of that information to be tracked, here (instead of relying 
> >>on an offshore vendor who may decide to copy IP). 
> >> 
> >>Aside from installing the above, the only real "mechanical" 
> >>modifications are connecting modules together and packaging. 
> >> 
> >>So, a "final test" can just verify proper operation of 
> >>the *device* (instead of its constituent modules). 
> >> 
> >>If modules are burned-in and tested prior to acceptance, domesticly, 
> >>the number of failures after assembly should be minimal. (Yet, 
> >>you'd still want to verify proper operation as the cost to repair 
> >>or replace far exceeds the price of the devices) 
> >> 
> >>OTOH, an assembly problem that could manifest during burn-in 
> >>would want some assurances that it wouldn't sneek past final 
> >>inspection in the absence of a post-assembly burn-in phase. 
> >> 
> >>At the very least, this sort of question would apply to anyone who 
> >>installs firmware after offshore manufacturing. Or, assembles 
> >>subsystems sourced offshore. 
> >> 
> >>So, what is the best practices guidance? 
> > 
> >We were doing overnight burnin+test of our products, after automated 
> >test and cal, but the failure rate was zero. Useful burnin might take 
> >weeks and temperature cycling or something expensive like that. 
> > 
> >Temperature cycling is probably the biggest stressor of parts and 
> >solder joints and design margins. Shock+vibration next. Just benign 
> >burnin doesn't seem to do much.
> Yes. See Accelerated Life Cycle Testing: 
> 
> .<https://en.wikipedia.org/wiki/Accelerated_life_testing#:~:text=Accelerated%20life%20testing%20is%20the,a%20short%20amount%20of%20time.> 
> 
> Joe Gwinn

That article is typical of what you find in the reliability engineering literature, a worthless collection of definitions, completely unusable garbage.

NASA has the most reasonable approach- keyword is approach.
https://sma.nasa.gov/sma-disciplines/eee-parts

On Friday, December 2, 2022 at 6:22:48 PM UTC-5, John Larkin wrote:
> On Fri, 2 Dec 2022 15:09:26 -0800 (PST), Fred Bloggs 
> <bloggs.fred...@gmail.com> wrote: 
> 
> >On Friday, December 2, 2022 at 1:08:08 PM UTC-5, John Larkin wrote: 
> >> On Fri, 2 Dec 2022 10:53:51 -0700, Don Y <blocked...@foo.invalid> 
> >> wrote: 
> >> >We're making devices that are typically built from 3+ "modules" 
> >> >(form factor is highly constrained so board space is similarly). 
> >> > 
> >> >Presently looking at the cost-benefit assessment of burning-in 
> >> >the modules and/or assembled devices. 
> >> > 
> >> >*Modules* will be made offshore so any test/burnin has to be part 
> >> >of the quoted manufacturing cost. 
> >> > 
> >> >Modules will be "post processed" domestically to install final 
> >> >firmware, S/Ns, private keys and watermarks. This allows for 
> >> >all of that information to be tracked, here (instead of relying 
> >> >on an offshore vendor who may decide to copy IP). 
> >> > 
> >> >Aside from installing the above, the only real "mechanical" 
> >> >modifications are connecting modules together and packaging. 
> >> > 
> >> >So, a "final test" can just verify proper operation of 
> >> >the *device* (instead of its constituent modules). 
> >> > 
> >> >If modules are burned-in and tested prior to acceptance, domesticly, 
> >> >the number of failures after assembly should be minimal. (Yet, 
> >> >you'd still want to verify proper operation as the cost to repair 
> >> >or replace far exceeds the price of the devices) 
> >> > 
> >> >OTOH, an assembly problem that could manifest during burn-in 
> >> >would want some assurances that it wouldn't sneek past final 
> >> >inspection in the absence of a post-assembly burn-in phase. 
> >> > 
> >> >At the very least, this sort of question would apply to anyone who 
> >> >installs firmware after offshore manufacturing. Or, assembles 
> >> >subsystems sourced offshore. 
> >> > 
> >> >So, what is the best practices guidance? 
> >> We were doing overnight burnin+test of our products, after automated 
> >> test and cal, but the failure rate was zero. Useful burnin might take 
> >> weeks and temperature cycling or something expensive like that. 
> >> 
> >> Temperature cycling is probably the biggest stressor of parts and 
> >> solder joints and design margins. Shock+vibration next. Just benign 
> >> burnin doesn't seem to do much. 
> > 
> >Infant Mortality--The Lesser Known Reliability Issue 
> >https://ieeexplore.ieee.org/document/4274831
> Paywalled. What's a reasonable burn time to catch infant mortality? If 
> it's months, it wouldn't be practical for commercial gear.

It's not the only game in town. There's tons literature on infant mortality with testing techniques dating to WW2.

On 12/2/2022 3:40 PM, Joe Gwinn wrote:
> Yes.  See Accelerated Life Cycle Testing:
> 
> .<https://en.wikipedia.org/wiki/Accelerated_life_testing#:~:text=Accelerated%20life%20testing%20is%20the,a%20short%20amount%20of%20time.>

You use accelerated testing to stress the *design*/process.
You want to identify the "weak points" in both and determine
if they are at acceptable levels for your market.

For each set of UNacceptable failures, you revise the design/process
and repeat the test.  I.e., you are deliberately BREAKING units to
identify how/when they fail(ed).

Eventually, you've quantified the shape of the bathtub curve
and the points where the "infant mortality" and "wearout" knees
occur.

Burn-in is a deliberate attempt to "age" the unit to the point beyond
the infant mortality knee -- on the assumption that it will then
be reliable to the wearout knee.

     Manufacture -> burnin -> test -> use

The goal is to prevent units that would have failed during the IM
period from progressing to the "use" (state) state.

If, for example, your design/process was flawed to the extent that
it merited an additional "test" stage between "manufacture" and
"burnin", then you should revise the design/process to eliminate
that extra handling stage.  I.e., so you have a reasonably high
degree of confidence that the items were *manufactured* correctly
and you now just need to stress the item before sale.

Once you've identified the knees for your design/process, you can
*sample* production to verify these criteria haven't changed
(or, that your process hasn't subtly changed).  This sampling
is as destructive as the initial assessment of the design/process
so you don't do it on every manufactured unit ("Yup!  This fuse failed
at the same point as the last one!")

We are addressing three different types of markets (consumer,
commercial/institutional and industrial).  We would *like* the
"modules" to be processed the same, offshore.  Otherwise, we
have to treat effectively identical "parts" as if they were
different parts -- because they were stressed differently.

And, the domestic "test" activities can be costly:  E.g.,
industrial parts are tested WHILE on the shaker table (because
shaking them and THEN testing isn't the same; they have to
operate WHILE being subjected to vibration).  Consumer
devices have to be tested at wider operating ranges than
industrial, etc.

And, until the modules are assembled into a "device", you
can't pronounce the end product as "tested" (a mechanical
assembly responds differently to vibration than a little
"board"; FLASH contents may degrade at elevated temperatures;
etc.).

We're not concerned with "fixtures" as the *devices* can be
used to test DUTs.  They already have the ability to perform
exhaustive self-testing after deployment so this can be
used to convert them to fixtures to test "unknowns".

This approach lets us make as many test fixtures as we want
just by reimagining "devices" as "testers".  E.g., the power
supply module can be made programmable so the power delivered
to the DUT can be "stressed", etc.  And, because each "device"
has a Gbe NIC, the number of test stations can easily scale
without having to worry about inherent limits to a tester
implementation -- test one DUT per "tester"!  (wanna test more
than 100 units at a time?  Add another switch; the additional
testers bring the processing power they need *with* them!)

It also gives us the "hook" to install the firmware, keys,
etc. without having to create yet another "programmer"
(and the NICs on the testers means they can interact with
the database that tracks all of this).

On Fri, 2 Dec 2022 15:38:08 -0800 (PST), Fred Bloggs
<bloggs.fredbloggs.fred@gmail.com> wrote:

>On Friday, December 2, 2022 at 6:22:48 PM UTC-5, John Larkin wrote:
>> On Fri, 2 Dec 2022 15:09:26 -0800 (PST), Fred Bloggs 
>> <bloggs.fred...@gmail.com> wrote: 
>> 
>> >On Friday, December 2, 2022 at 1:08:08 PM UTC-5, John Larkin wrote: 
>> >> On Fri, 2 Dec 2022 10:53:51 -0700, Don Y <blocked...@foo.invalid> 
>> >> wrote: 
>> >> >We're making devices that are typically built from 3+ "modules" 
>> >> >(form factor is highly constrained so board space is similarly). 
>> >> > 
>> >> >Presently looking at the cost-benefit assessment of burning-in 
>> >> >the modules and/or assembled devices. 
>> >> > 
>> >> >*Modules* will be made offshore so any test/burnin has to be part 
>> >> >of the quoted manufacturing cost. 
>> >> > 
>> >> >Modules will be "post processed" domestically to install final 
>> >> >firmware, S/Ns, private keys and watermarks. This allows for 
>> >> >all of that information to be tracked, here (instead of relying 
>> >> >on an offshore vendor who may decide to copy IP). 
>> >> > 
>> >> >Aside from installing the above, the only real "mechanical" 
>> >> >modifications are connecting modules together and packaging. 
>> >> > 
>> >> >So, a "final test" can just verify proper operation of 
>> >> >the *device* (instead of its constituent modules). 
>> >> > 
>> >> >If modules are burned-in and tested prior to acceptance, domesticly, 
>> >> >the number of failures after assembly should be minimal. (Yet, 
>> >> >you'd still want to verify proper operation as the cost to repair 
>> >> >or replace far exceeds the price of the devices) 
>> >> > 
>> >> >OTOH, an assembly problem that could manifest during burn-in 
>> >> >would want some assurances that it wouldn't sneek past final 
>> >> >inspection in the absence of a post-assembly burn-in phase. 
>> >> > 
>> >> >At the very least, this sort of question would apply to anyone who 
>> >> >installs firmware after offshore manufacturing. Or, assembles 
>> >> >subsystems sourced offshore. 
>> >> > 
>> >> >So, what is the best practices guidance? 
>> >> We were doing overnight burnin+test of our products, after automated 
>> >> test and cal, but the failure rate was zero. Useful burnin might take 
>> >> weeks and temperature cycling or something expensive like that. 
>> >> 
>> >> Temperature cycling is probably the biggest stressor of parts and 
>> >> solder joints and design margins. Shock+vibration next. Just benign 
>> >> burnin doesn't seem to do much. 
>> > 
>> >Infant Mortality--The Lesser Known Reliability Issue 
>> >https://ieeexplore.ieee.org/document/4274831
>> Paywalled. What's a reasonable burn time to catch infant mortality? If 
>> it's months, it wouldn't be practical for commercial gear.
>
>It's not the only game in town. There's tons literature on infant mortality with testing techniques dating to WW2.

Military electronics used to require JAN/TX transistors, carefully
assembled and tested and burned in and fabulously expensive. Then
someone determined that regular equivalents were more reliable.