Author Topic: End of an era  (Read 659 times)

0 Members and 1 Guest are viewing this topic.

Offline MadScientist267

  • Impossible Condition Curator
  • Moderator
  • Hero Member
  • *****
  • Posts: 1514
  • Karma: +44/-4
  • Rules? What rules?
End of an era
« on: December 20, 2018, 12:42:47 pm »
Well, it's over.

At least for the lead.

A series of events and concepts came together in something of a perfect storm, and the truck batteries paid the price.

Most here already know that lead acid is a little on the particular side when it comes to what they need to keep them healthy...

In the truck especially, it was a constant battle to try and keep them happy. They were subject to all the typical abuse solar just naturally presents, and despite being appropriately sized originally, they ended up being subject to some heavier loads than intended (namely the air conditioner and microwave)... And while they could support them at the transient level, the impact of doing so combined with the timing needs (evening mostly) left them repeatedly sitting at partial charge for hours on end with the base load still to do it's thing.

Another regular factor that compounded all this, vibration in transit isn't like what they experience in a golf cart... Video shot back there once for grins revealed something resembling a non trivial earthquake.

Then there's temp extremes, anything from about -10C to 40C ambient exposures on occasion, with 15-30 being more typical, not to mention the internal heat generated during absorb. The peltier system worked surprisingly well with the exception of causing a gradient across the box that led to regular imbalance tendencies, and because of this, their effectiveness was limited more by the amount of power the box could thermally tolerate than the actual capability or efficiency of the peltier system itself.

Not last and certainly not not least, the early balance and charging issues on the block 3/4 pair due to a defective crimp that went undetected for quite a while in the very beginning...

I digress. You get the idea. They saw some $#!+ in a relatively short period of time.  :-\

The final straw was the timing of the solar field job and the idea that I couldn't get to the truck to move it up to its "winter spot" so it could get adequate light for about a month beyond where the sun was starting to hide for too long behind trees. By the time I got there, the graphs weren't painting a pretty picture, but it was still holding out surprisingly well considering.

I believe my next fatal mistake was in a time crunch to try and squeeze in what needed to happen, I set up the next part of the stage for failure.

I went for an hour long exercise run at 60 MPH when they were in the low 80% SoC range (that's as reported by the classic... I believe actual was probably closer to mid or even low 70s)...

They had also received a dose of water, although done differently than I usually do... I fed them by known volume rather than to a level. I don't believe the watering played any real part in the failure directly... But I mention it because they were on the thirsty side and while the plates were all still covered, the SG would've been on the elevated side, aiding in degradation.

Granted, I had started the genset before beginning the run, but I knew that wasn't really going to help save them from the physical trauma they were about to endure... Only helped increase the efficiency of my use of time. The plates were in a weakened state, and due to the low rates of charge and discharge for "deep sleep" mode, large sulfate crystals were present... One of which was right at the top of one plate clearly visible with the cap off. It severely deformed the separator next to it, and while I knew that specific one wasn't posing any threat to the cell, I knew there had to be more like it elsewhere.

When I got back from the ride, it was gone... With the appearance that it had broken off and gone elsewhere. Neck hair stood up further, but what could I do at that point, right?

Continuing the charge from genset, the graphs were beginning to paint a picture that they were not happy with me at all, but I finished out to very close to "flat net" and knowing the next piece of the puzzle was likely coming, I called them full and crossed my fingers.

A few days later, while I was once again unable to get to it to do anything, a snow storm hit the area, and while only depositing about 2.5", as anyone that's dealt with it knows, snow is effectively an "indefinite midnight" for PV... And with temps not being conducive to a melt, it effectively sat in the dark for a week before I even got up there to find out that it had gone completely wrong, nevermind the mode of failure.

Not enough? I managed to forget the key to get into the back so I could take a closer look at what had happened, and my usual "check from the phone" thing wasn't happening.

No wifi SSID.


The only other thing I could detect from outside at that point was that the fresh air intake fan had also quit running... My stomach sank... It would still be a couple days before I could get back with the key to find out more.

When I finally did get inside, I was greeted with pitch black except for a single faintly glowing red pilot LED that was part of the DC outlet ("cigarette lighter") trio... Nothing else.

I carefully checked a few things to make sure I didn't disturb a flaky disconnect or anything like that, even tho I had pretty much ruled that out at this point. After laying the meter across the bus and seeing 2.475V... there was not much left to do but disengage the latches on the battery box, open it up, and make sure they hadn't gone completely stupid on me for whatever reason. They hadn't... But I was quickly realizing that it was over. One more thing to do, just in case.

I filled the genset up and got it warming up, and went inside to disconnect the Pi because what I was about to attempt, I hadn't done yet. Ever.

I pulled the master battery disconnect key out, started up the charger, and attempted to set it up in power supply mode. This indeed worked at one point in time, but I knew the modifications made after this had been tested could possibly present an unstable output in the absence of the batteries to provide "flywheel" for it. Sure enough, the first voltage I saw on the truck's permanent LED voltmeter was 19.5...

I immediately killed the charger's grunt mains disconnect, but not before hearing something to my right pop inside (what I believe was) one of the boxes on the aux power board. From the sound of it, probably a small cap.

So I put the disconnect key back in, set things to a very conservative near nil on the charger, and began ramping it up slowly.

I never saw more than 7.0V on the voltmeter while trying to do a modest recovery charge limited to just over 11A. I let it sit for about a minute after it stabilized there... It wasn't looking to cooperate, and given what I knew already, I didn't want to risk anything going more stupid than it already had.

They're done.

What follows still needs verification if it's even possible... There's a chance the Pi cycled in the process of the brownout and there's no telling what may have come of that if so. Computers don't like that.

What I believe happened was that while the label capacity would have stood up to the week long absence of light when they were new, I don't think there was enough left of them to handle it at this point.

The draw in deep sleep (with the intake fan running) is about 1.2A... the bank's label is 430Ah. No they wouldn't have been happy about it and would have sustained some damage, but also wouldn't have completely given up and gone "below 0%" if they'd have been up to it.

The master blow that sealed their fate was that the classic (charge controller) eventually saw a lower operating voltage limit, and because the deep sleep load is so low anyway, there was likely very little if any rebound, and so effectively locked itself out. For the record, despite some "hard feelings" toward Midnite Solar over unrelated issues early on in the design phase, I want to make it clear that this isn't a "dump" on them... The behavior was expected.

What it meant tho was that until they could no longer hang on, the few loads that run in deep sleep mode dropped off one by one, burying the lead in the process.

The future of the truck as a whole and concept is uncertain at this point... It's been largely unattended for coming up on a year, as I've found my soul mate and life is moving on. That said, I was a bit upset that I wasn't able to take it offline personally... Which was the plan for Jan 15 2019... Marking 3 years since launch.

I intend to go rescue the SD card out of the Pi rather than trying to light it up in situ, largely because I hope to also capture the last "diagnostics page" that could still, possibly, contain the number for the stellar uptime I was able to achieve.

In the process, again assuming it didn't scramble itself thanks to the marginal supply, I hope to be able to pull up all the graphs and get confirmation one way or the other on how exactly the horror unfolded. If they're present, it'll be anything but "just another day in the truck" , and I'll post the pertinent ones here for all to see.

Thanks again to everyone who helped make this happen... It was a lot of fun, I learned more than I'll ever be aware of even I'm sure in the process, about the technologies, nature, and myself.

And I don't regret doing it even for a moment.

Until next time...

Wanted: Schrödinger's cat, dead and alive.

Offline bj

  • Hero Member
  • *****
  • Posts: 728
  • Karma: +23/-0
  • Lamont, Alberta, Canada
Re: End of an era
« Reply #1 on: December 22, 2018, 05:16:08 am »
  Too bad.  In many ways.
  But it was part of a journey that brought you to where you are today.
  From posts over the years, and reading between the lines, It seems that you
are a much happier person, so I would label the project hugely successful.
  Best of luck on the next part.
"Even a blind squirrel will find an acorn once in a while"

Offline MadScientist267

  • Impossible Condition Curator
  • Moderator
  • Hero Member
  • *****
  • Posts: 1514
  • Karma: +44/-4
  • Rules? What rules?
Re: End of an era
« Reply #2 on: December 22, 2018, 01:04:05 pm »
Thanks BJ... and yes, it has changed me in several ways... Many I'd like to think of as being for the better.

Lots of ideas on the table at the moment... No idea which ones are going to stick, but you can bet my journey continues on one way or the other... Always lol
Wanted: Schrödinger's cat, dead and alive.

Offline WooferHound

  • Technowhiz
  • Global Moderator
  • Hero Member
  • ******
  • Posts: 897
  • Karma: +40/-3
  • Huntsville Alabama USA
    • My personal webpage
Re: End of an era
« Reply #3 on: December 24, 2018, 06:27:30 am »
Is the problem just the Batteries, or are there other problems holding you back. Will New Batteries fix the problem.

I feel your pain after retiring my last set of batteries a coupla months ago, spending a day trying to recover them.
And now my silly little small tractor battery is getting noticeably weaker every day. I will be getting some kind of replacement for them within the next month, even if they are used ordinary car batteries.
----- W o o f e r h o u n d -----
My Renewable Energy Projects

Offline MadScientist267

  • Impossible Condition Curator
  • Moderator
  • Hero Member
  • *****
  • Posts: 1514
  • Karma: +44/-4
  • Rules? What rules?
Re: End of an era
« Reply #4 on: January 07, 2019, 01:37:18 am »
Mainly just the batteries Woof... Still no idea what exactly it was that I popped when I was trying to bring it up on only the charger... But didn't sound big. Just would like to know what and where it was.

I did recover the SD card from the Pi when I was there not long ago, but forgot how much of it was set up to run in RAM and so the graphs, while most likely present, are in an archive and may be missing as much as an hour worth of data processing just prior to the actual disconnect. The file system seems intact, just a matter of pulling everything out of the archives used for protecting the data from power loss like it experienced. I'll put it up as soon as I get the chance... Lots of other things going on at the moment as well.

Wanted: Schrödinger's cat, dead and alive.

Offline MadScientist267

  • Impossible Condition Curator
  • Moderator
  • Hero Member
  • *****
  • Posts: 1514
  • Karma: +44/-4
  • Rules? What rules?
Re: End of an era
« Reply #5 on: January 09, 2019, 12:19:46 am »
Ok well here goes... Part 2 of it anyway.

The initial comb thru the file system left me feeling a little like an alien in my own world... But once I scraped the rust off... I was able to piece together the last time slice worth of data at least, as the graphs need still need to be reconstructed from the data structure that they're built from.

First and foremost, the theory that the classic had already shut down and imminent doom had set in before the Pi dropped offline has been disproven at this point.

It was shown to have been still responding to polls, accurately time stamped, based on data from the last parachute backup: Dec 12 at 8AM.

This data set may precede the actual protective disconnect by up to 59 minutes before it's final breath...

1. Battery voltage reported by the classic matched with readings from the battery monitor's ADC

2. Classic had a PCB temp well above ambient temp

3. No polling errors related to the classic in the logs

Battery monitor system reports:

Pair A:
Block #1: 5.260V
Block #2: 3.156V
Total: 8.416V

Pair B:
Block #3: 5.442V
Block #4: 2.956V
Total: 8.398V

Classic reports:

Mode state reported was: Resting

Battery voltage reported was: 8.3V.

Net current reported by WBJR was: -2.9A

8.3V at 2.9A indicates a load of about 24W. That's about right with the intake fan running and everything else in deep sleep mode for minimal system operation, with little or no solar input.

It was normal to see the classic report slightly under the battery monitor under load conditions (and slightly higher during charging)

PCB temp reported was: 17.9C
FET temp reported was: 8.4C

FET temp being below board temp by this much also indicates very little if any production by the PV, consistent with the time of day and expected negative net current being drawn from the battery.

Pyranometer report:

The pyro shows it was receiving light, adding a bit of insult to injury.

Pyro reported insolation: 27.675 W/m²

It would appear that by 8AM, the snow was gone at least off of the sensor. This doesn't necessarily indicate the entire roof was clear but at least *some* light would have been able to theoretically reach at least some of the PV. Exact sky conditions at that moment are still unknown at this point.

Temperatures reported:

Ambient outside temp reported: 7C
Ambient inside temp reported: 1.625C
Thermal Mass discharge temp: 0.75C

Temperatures had at least become favorable for melt outside, even if there was no direct light to assist this. Going by the lag in both inside ambient and thermal mass probes, conditions just prior to this time slice were more conducive to maintaining the freeze than helping with melt.

Battery Health:

SoC reported by classic was: 45%

With the voltages as low as they had become, this indicates significant degradation to the batteries. They really HAD taken a beating

I knew they were in rough shape but didn't expect they'd been reduced to only about half of label... Not exactly surprised, just didn't estimate things had progressed that far.

I figure a fair chunk of it probably happened in the last month prior to being able to move it to get better light... it was sitting mostly shaded. It's of course difficult to say how much damage was done when, as they served a hard life in normal operation, in addition to the extra abuse at the end caused by the shade.

A little bit of feel good story to tack on to the end... Sorta...

top - 13:30:50 up 367 days

So between 8:30 and 8:35AM... The Pi's power supply made a clean break.

(Side note - The RTC scheme used had the classic operating as the master clock, which was set to GMT. The Pi used this time as well, the corrections were made for local in the graphs on the fly)

That really sucks... Couple more hours and I figure more than likely it would have actually survived. For how much longer, well of course, anyone's guess.

It also could have been a bit worse...

One more tidbit I caught just before wrapping up this part of the investigation...

So used to seeing "meaningless" crap in the log tail that I didn't scrutinize it closely at first...

The final straw for the Pi may have been the actual alarm itself tipping it's power that last little bit...

I show 3 recorded poll failures from the ADC that pissed python off, probably because the regulators on the ADC couldn't hold 5V anymore

So for about 15 minutes before the shutdown, the alarm LED was pulsing... The +5V feed for the LED (1W red) being taken from the Pi's header

The supply feeding the Pi was no slouch, a full blown micro PC DC-DC supply complete with crowbar... There was a reason for that, even tho this wasn't exactly one of the scenarios envisioned when it was selected.

I'm just glad it worked out as "designed". I fear with great certainty that in this situation, a simple buck converter supply would have likely wrecked shop because it wouldn't have shut itself down in paranoia and the pi could have fringe cycled it's way into corrupting the SD card. Granted it was designed at the software level to do minimal writing... But that's only on my end. The OS was standard Raspbian Wheezy, which I know to a point is tweaked for SD but I don't know to what extent, and hadn't really done any tweaking of my own on it for those purposes.

Either way, all the data appears to be intact... It's just not in a very friendly form to externally recover and directly view.

The next fun part is setting up a dummy run of ranges with the date/time forced so it generates the graphs as were at time of death... Then I can post the pretty pictures of what only I can "see" at the moment in my mind, as well as the entire "bigger picture".

There's no doubt that at some point things just had gone too far and the classic gave up, as suspected. The order this took place in however, a bit of a surprise. If I'm right, there won't be any "flat" lines on the data points that come from the classic. It also means that nobody will ever know exactly when the classic gave up and threw the batteries, once and for good, up under the... Truck.

Guess we'll see, won't we. I'll get to it as soon as I can, there's a lot going on at the moment that takes precedence.

Till then...
Wanted: Schrödinger's cat, dead and alive.

Offline rossw

  • Senior Moderator
  • Hero Member
  • *******
  • Posts: 876
  • Karma: +35/-0
  • Grumpy-old-Unix-Admin
Re: End of an era
« Reply #6 on: January 09, 2019, 04:46:24 am »
Nice work getting all that after the event, Steve.
Doesn't change things of course, but I'm sure you'll have some amount of pride in knowing you worked out how it all went down.