BE copy to non booting mirrored drive


#1

Hi all,

My TrueOS system has two ZFS pools, both mirrored, one for the O/S and one for /home. It is now on 12-CURRENT, 7/16 version.

After the last update today I could not boot from the primary drive, but it does from the 2nd drive. Now I’m trying to figure out how to copy (clone?) the BE from the secondary to the primary drive but am not able to find the proper search terms or something.

Any pointers?


#2

I’ve verified the primary drive and it does not appear to be a matter of drive corruption but rather looks like not having received the BE update, or able to complete writing it.

Upon boot it gives the bios options and clears the screen with a blinking cursor in the upper left corner. (The same behavior you get if you insert, for example, a non-bootable DVD which it then tries to boot from and simply hangs. You would expect it to instead move on and trying to boot from another drive, but mine has never done that, maybe a bios setting…)

My idea is that it should be able to mirror the 2nd drive onto the first by noticing the time stamp is newer on the 2nd drive in the mirror. But I’m simply not familiar enough.


#3

Not really sure, but let’s give or a whirl.

Can you see the original boot drive? If so, can you select it to roll back to a previous BE?


#4

I can see it as part of the mirror, but have no idea on how to select and then roll back. On my mirror both drives are fully identical, but now for the BE appearing to be missing.
I’m only able to boot from the second drive in the mirror. Booting of the first one is not an option. I have to go through BIOS to tell it to boot from the other drive.


#6

Maybe @mer might have an answer to tip on this.

Don’t want to screw up a working system


#7

“I’ve verified the primary drive and it does not appear to be a matter of drive corruption but rather looks like not having received the BE update, or able to complete writing it.”

Since the storage-pool is configured to be a mirror, then by definition, the entirety of the BE update must be on all disks which make up the vdev unless the pool became degraded. Which in your case could be a failing primary-disk (a.k.a. your boot disk configured in BIOS) and if so maybe the pool is still in a degraded state.

To start with, check the “health” of the pool:
zpool status -x

If the pool is healthy, then I don’t know how the new BE was only written to one of the disks in the mirror. Like I said, that goes against the definition of mirrored storage. I suspect the pool isn’t healthy and the primary disk is failing. If so, take the primary disk offline to start with:
zpool offline

After the device is off-line, zfs will no longer try to use it.

I suppose that the next step(s) would be to detach it and physically replace it.
Here’s a guide on how to do that:
https://docs.oracle.com/cd/E19253-01/819-5461/gbcet/index.html


#8

start with what @fjs asks, only do it without the -x that will help us determine that zfs really thinks the drives (vdevs) should be mirrored and that they are in sync.
Another possibility is that for some reason one of the drives lost it’s “bootability”.
gpart list to let us see everything.
Is this UEFI or legacy/CSM boot? An unlikely possibility is that the boot partition lost the bootloader files.


#9

Sorry, forgot to mention that ZFS is reporting the pool healthy. I ran an extended SMART test and it also says good, in fact nothing looks bad.

I understand that as mirrored drives they should be identical. Clearly something is going on. Resilvering might be a way to go, but I’m not sure how I can guarantee which will be the source.

gpart also says all is ok.

This is a gen 9 supermicro so no UEFI. Controller is off the m/b without any RAID.

Lost bootloader files sounds like a simple start. Can those simply be copied over or do they need to be modified?


#10

gpart bootcode command should work.
gpart list what we want to make sure of is that the 2 boot devices are partitioned the same, so providing that output would be nice, along with telling us which 2 are the mirrored boot devices.
the first partition on each should be type freebsd-boot, at least 256K (1MB is probably better).
To put the bootcode on the partition do something like NOTE this is for a legacy/BIOS boot. Your freebsd-boot type partition is probably partition 1 (the -i argument) substitute the correct devices for ada0. If you know which one it won’t boot from you can just do it to that device.

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

EDIT: I had a typo in the above command, forgot the leading / on the -p arg

edit:
resilvering (basically the 2 vdevs resyncing) means the 2 parts of the ZFS datasets are in sync. Really doesn’t have anything to do with booting.
If both devices are showing up in the system as part of the correct mirror and they are not showing errors and are in sync, they are consistent at the ZFS dataset level. I would do a zpool scrub on it to force everything to be consistent, then do the gpart bootcode command then reboot


#11

gpart bootcode, got it.

They are two SSD’s about 240G.

Partitions were created by the installer, I simply told it to make a mirror of two drives. ada0 is the not booting one and ada1 is bootable.

Would it not be favorable to run gpart first and see if it solve the problem?


#12

Yes, that’s what I am suggesting. All I was asking for was to provide some information from the gpart list command so I could make sure I was telling you the right stuff. I’m going to assume that the installer partitioned the devices correctly and the same so if ada0 is not booting then I would try entering this command:

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

that will rewrite the bootloader stages into the first gpart partition on device ada0.

The zpool scrub is a personal preference; I manually run it once every 3 or 4 months. If the booted system, zpool status shows ada0 and ada1 together in a mirror and they are both online, doing a scrub will ensure that the ZFS data on them is consistent.
zpool scrub tank to start it
zpool status will show the current progress
eventualy zpool status will show the scrub completed.

But no, you don’t have to run zpool scrub. I’m just trying to provide complete help in the best manner I can.


#13

Thank you! Yes I figured that was what you were doing but I’d rather double check! :slight_smile:

I’ve not bothered with the scrub, or even considered it a practical thing. Probably putting too much reliance on ZFS. :slight_smile:

How long time can I expect it to take on something like 250G SSD? Or if that is too vague is it a fast process in the matter of an hour or days?

BTW:
partcode written to ada0p1
bootcode written to ada0

Now I will reboot.


#14

Ahh, it’s not tied to the size of the device. It’s tied to how much is in use by ZFS. I’ve got a couple of different systems with SSDs (not mirrored) It takes about 5 minutes or so for about 35-40G on my SSDs.
A mirror pair on WD RED drives takes about 20 - 30 mins. for about 60G for me.
How long is directly tied to “how much” how much data, how much bandwidth for the hardware interface to the devices. NVMe will be faster than SATA3 SATA3 faster than SATA2, typically SSD will be faster than spinning drives.

EDIT
scrub is like opinions, everyone has one. ZFS has checksums of pretty much every data block and metadata structure. What scrub does is run through all of that recomputing checksums and correcting data or checksums. It gets interesting on mirror pairs and RAIDZ configurations.
How often to run depends on the quality of your hardware and the importance of your data. My opinion (based solely on things I’ve read, the systems I have (had)) is run it less often on commercial grade high reliability devices, more often on consumer grade devices. Some like to run it at least once a month, others once a year. I try to run it around every 3 or 4 months. You can still use the system while it’s going on (not like fsck) you get slightly degraded performance.


#15

OK, thanks, good to know! Scrub basically does what ZFS does every time you access a file but for all files.

As you saw above the gpart command executed successfully but would not boot. I had to use BIOS to tell it to boot the 2nd drive. (Handy to be mirrored!)

So scrub is not going to repair anything but bad checksum files. Not knowing now what is off it seems to me that I now should resilver, do I have that correct?

Would it be enough to export and then import it back for it to notice the discrepancy and “fix” it? And finally how do I ensure the correct drive is the source?


#16

Not sure but export and import shouldn’t hurt.
If the system is currently booted off of ada1, and when the system is booted it shows the mirror pair ok, and you are in the correct BE (beadm list, look for where the N and R show up under the Active column) then I really don’t think the issue is at the ZFS level.
You say the system boots off ada0 when you force it to or does it not boot off of ada0 at all? If not at all then it’s the BIOS is not recognizing ada0 as a bootable drive which is usually down in the partitioning somehow.
Difficult to say how to fix it.


#17

Yes, in order to boot I need to tell BIOS to boot of ada1. Using ada0 it simply hangs the boot process at the point when it looks for a drive. (Flashing cursor top left after BIOS screen goes away.)

It must be doable to re-mirror the drive, no?

Hopefully this is not too much data:

gpart list

Geom name: ada0
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 488397127
first: 40
entries: 152
scheme: GPT
Providers:

  1. Name: ada0p1
    Mediasize: 262144 (256K)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r0w0e0
    efimedia: HD(1,GPT,e8b91e4b-c9b7-11e6-afee-0025902dffd4,0x28,0x200)
    rawuuid: e8b91e4b-c9b7-11e6-afee-0025902dffd4
    rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
    label: (null)
    length: 262144
    offset: 20480
    type: freebsd-boot
    index: 1
    end: 551
    start: 40
  2. Name: ada0p2
    Mediasize: 235746099200 (220G)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(2,GPT,ea1ea625-c9b7-11e6-afee-0025902dffd4,0x228,0x1b71c800)
    rawuuid: ea1ea625-c9b7-11e6-afee-0025902dffd4
    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
    label: (null)
    length: 235746099200
    offset: 282624
    type: freebsd-zfs
    index: 2
    end: 460442151
    start: 552
  3. Name: ada0p3
    Mediasize: 4294967296 (4.0G)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(3,GPT,eb86acd1-c9b7-11e6-afee-0025902dffd4,0x1b71ca28,0x800000)
    rawuuid: eb86acd1-c9b7-11e6-afee-0025902dffd4
    rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
    label: (null)
    length: 4294967296
    offset: 235746381824
    type: freebsd-swap
    index: 3
    end: 468830759
    start: 460442152
    Consumers:
  4. Name: ada0
    Mediasize: 250059350016 (233G)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r2w2e4

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 468862087
first: 40
entries: 152
scheme: GPT
Providers:

  1. Name: ada1p1
    Mediasize: 262144 (256K)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r0w0e0
    efimedia: HD(1,GPT,e6e2bcae-c9b7-11e6-afee-0025902dffd4,0x28,0x200)
    rawuuid: e6e2bcae-c9b7-11e6-afee-0025902dffd4
    rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
    label: (null)
    length: 262144
    offset: 20480
    type: freebsd-boot
    index: 1
    end: 551
    start: 40
  2. Name: ada1p2
    Mediasize: 235746099200 (220G)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(2,GPT,ea186ec0-c9b7-11e6-afee-0025902dffd4,0x228,0x1b71c800)
    rawuuid: ea186ec0-c9b7-11e6-afee-0025902dffd4
    rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
    label: (null)
    length: 235746099200
    offset: 282624
    type: freebsd-zfs
    index: 2
    end: 460442151
    start: 552
  3. Name: ada1p3
    Mediasize: 4294967296 (4.0G)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(3,GPT,eb803882-c9b7-11e6-afee-0025902dffd4,0x1b71ca28,0x800000)
    rawuuid: eb803882-c9b7-11e6-afee-0025902dffd4
    rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
    label: (null)
    length: 4294967296
    offset: 235746381824
    type: freebsd-swap
    index: 3
    end: 468830759
    start: 460442152
    Consumers:
  4. Name: ada1
    Mediasize: 240057409536 (224G)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r2w2e4

#18

I agree with @mer that at the ZFS level everything is ok. So, I don’t think that “remirroring” the ada0 disk will fix the problem. For some reason, seems like the BIOS just can’t find or can’t read the boot-sectors on ada0 even though you just reinstalled the boot-code. I don’t know why.


#19

An extremely brute force way would be to unmirror ada0 and ada1, blow away all partitioning on ada0 (gpart destroy -F), repartition (if the devices are the same an easy way would be to do
gpart backup ada1 | gpart restore -F ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

then readd ada0p2 to the mirror. But it still may not fix the BIOS can’t see it as a boot drive. Is ada0 actually seen in the BIOS?
But i


#20

Yes, ada0 is visible in BIOS and selectable as a boot device. Choosing it will only produce a blank screen with a blinking cursor.

It is also visible from gpart once booted from the other drive.


#21

blinking cursor, what shape? a straight line, like the one that usually spins as the boot loader loads?

Edit:
the freebsd-boot type partition is typically not mounted (ada0p1 and ada1p1), which is why we use gpart to write the bootcode to it (-p parameter). There may have been an issue a long time ago where gpart bootcode appeared to work when you gave it both arguments but it actually didn’t.
How about trying a two step?

gpart bootcode -b /boot/pmbr ada0

Reboot and see if that fixed it. If not, after rebooting from ada1, do:
gpart bootcode -p /boot/gptzfsboot -i 1 ada0

and try rebooting from ada0.