TrueOS Update Problem 17.12 to 18.02


#1

Continuing the discussion from TrueOS Vulnerability Management / TrueOS CERT/CSIRT and ΔCERT:

After switching from STABLE to UNSTABLE track we rebooted our test VM. After that we updated the system and received the error messages below (“Failed Updating!” and “Rolling back…”). Can someone help us with debugging, please? Thanks in advance!


TRON-DELTA.ORG – TrueOS PC-Updatemanager Security Review 2018
#2

go back to the good Boot Environment (BE).

in the terminal run
sudo pc-updatemanager pkgcheck
sudo pc-updatemanager pkgupdate
sudo shutdown -r now

and watch to make sure everything gets downloaded


#3

Just a minor tweak to what @RodMyers mentioned:

sudo pc-updatemanager pkgcheck
sudo pc-updatemanager pkgupdate
sudo pc-updatemanager startupdate

The “startupdate” flag is due to the newer “active” update system rather than being hooked into the shutdown routine at all times.


Please bring back my topic
#4

Would it be useful to add the “-f” to pkgupdate to force redownload of packages, just to make sure?


#5

% sudo pc-updatemanager
pc-updatemanager: Usage

branches - List available system branches
chbranch - Change to new system branch
check - Check for system updates
showeol - Show end of life (support) date for this release
install , - Install system updates
pkgcheck - Check for updates to packages
pkgupdate [-f] - Install packages updates (-f to force)
syncconf - Update PC-BSD pkg configuration
confcheck - Check PC-BSD pkg configuration
cron - Perform delayed check for system and pkg updates.

The parameter startupdate is not mentioned by the program. Will this be fixed in the next release or is the parameter experimental? Regarding the man page, we suggest to explain the the parameters in a bit more detail.

% sudo pc-updatemanager pkgcheck
Boot-strapping updater…OK
Updating the pkg database…
Updating trueos-base repository catalogue…
trueos-base repository is up to date.
Updating trueos-major repository catalogue…
trueos-major repository is up to date.
All repositories are up to date.
Checking for upgrades (17 candidates): … done
Processing candidates (17 candidates): … done
The following 17 package(s) will be affected (of 0 checked):

Strange: later in the process it says Number of packages to be fetched: 805 Are these dependencies?

Installed packages to be UPGRADED:
smartmontools: 6.6 -> 6.6_1 [trueos-major]
openjdk8: 8.152.16 -> 8.162.12 [trueos-major]
lsof: 4.90.q,8 -> 4.90_1,8 [trueos-major]
libxml2: 2.9.4 -> 2.9.7 [trueos-major]
libnghttp2: 1.28.0 -> 1.30.0 [trueos-major]
libffi: 3.2.1_1 -> 3.2.1_2 [trueos-major]
lha: 1.14i_7 -> 1.14i_8 [trueos-major]
json-c: 0.12.1 -> 0.13 [trueos-major]
java-zoneinfo: 2017.c -> 2018.c [trueos-major]
irssi: 1.0.5,1 -> 1.1.1,1 [trueos-major]
glib: 2.50.2_7,1 -> 2.50.3_1,1 [trueos-major]
expat: 2.2.1 -> 2.2.5 [trueos-major]
curl: 7.56.1 -> 7.58.0_1 [trueos-major]
clamav: 0.99.2_6 -> 0.99.3 [trueos-major]
ca_root_nss: 3.34.1 -> 3.35_2 [trueos-major]
arj: 3.10.22_6 -> 3.10.22_7 [trueos-major]
alsa-lib: 1.1.2 -> 1.1.2_1 [trueos-major]

Number of packages to be upgraded: 17

65 MiB to be downloaded.
The following updates are available:

NAME: System package updates
TYPE: PKGUPDATE

To install: “pc-updatemanager pkgupdate”

% sudo pc-updatemanager pkgupdate
Boot-strapping updater…OK
Boot-strapping latest PKGNG
Updating the package repo database…
Cleaning old pkg upgrade cache…
fetch -o /usr/local/pkg-cache/fbsd-distrib.txz http://pkg.cdn.trueos.org/unstable/amd64-base/fbsd-distrib.txz
/usr/local/pkg-cache/fbsd-distrib.txz 100% of 3586 kB 486 kBps 00m07s
/usr/local/pkg-cache/fbsd-distrib.txz.sha1 100% of 512 B 1358 kBps 00m00s
Verified OK
Updating trueos-base repository catalogue…
trueos-base repository is up to date.
Updating trueos-major repository catalogue…
trueos-major repository is up to date.
All repositories are up to date.
Verifying / fetching packages for ports-mgmt/pkg - pkg-1.10.5.txz

Number of packages to be fetched: 1
No packages are required to be fetched.
Is it safe to assume they and the other ones were downloaded with our last upgrade try?
Integrity check was successful.
Verifying / fetching packages for misc/trueos-desktop - trueos-desktop-201802271624.txz

Number of packages to be fetched: 405
No packages are required to be fetched.
Is it safe to assume they and the other ones were downloaded with our last upgrade try?
Integrity check was successful.
Determining package name for: editors/jedit
Verifying / fetching packages for editors/jedit - jedit-5.4.0,1.txz

Number of packages to be fetched: 34
No packages are required to be fetched.
Is it safe to assume they and the other ones were downloaded with our last upgrade try?
Integrity check was successful.
Determining package name for: irc/irssi
Verifying / fetching packages for irc/irssi - irssi-1.1.1,1.txz

Number of packages to be fetched: 10
No packages are required to be fetched.
Is it safe to assume they and the other ones were downloaded with our last upgrade try?
Integrity check was successful.
Determining package name for: security/clamav
Verifying / fetching packages for security/clamav - clamav-0.99.3.txz

Number of packages to be fetched: 11
No packages are required to be fetched.
Is it safe to assume they and the other ones were downloaded with our last upgrade try?
Integrity check was successful.
Determining package name for: security/rkhunter
Verifying / fetching packages for security/rkhunter - rkhunter-1.4.4.txz

Number of packages to be fetched: 4
No packages are required to be fetched.
Is it safe to assume they and the other ones were downloaded with our last upgrade try?
Integrity check was successful.
Determining package name for: sysutils/smartmontools
Verifying / fetching packages for sysutils/smartmontools - smartmontools-6.6_1.txz

Number of packages to be fetched: 1
No packages are required to be fetched.
Is it safe to assume they and the other ones were downloaded with our last upgrade try?
Integrity check was successful.
Verifying / fetching packages for FreeBSD Base

Number of packages to be fetched: 805
No packages are required to be fetched.

Is it safe to assume they and the other ones were downloaded with our last upgrade try?

Integrity check was successful.

Your update is staged and ready to install.
To reboot and begin the update run # pc-updatemanager startupdate

% sudo pc-updatemanager startupdate

After a reboot: Preparing to do pkg installation (58 pkgs)

The installation ran with success, but after a reboot the same message “Failed updating! Reverting to last good boot-environment…”

In a different post we found:

sudo pkg remove freebsd-examples
sudo pkg-static install -f freebsd-runtime

Would that be helpful in any way?

% sudo pkg_tree | grep 'examples’
FreeBSD-examples-12.0.s20180212150758

Also we did not check /var/log/pc-updatemanager.log, but would not expect more than the terminal output to be in the log, right?


#6

Check version:

% pkg-static --version
1.10.5

I had problems, too.
Now, it works.

But unfortunately, single user mode was neccessary. Experience was a bit unsatisfying…

Update pkg-static alone, first (if not present). May help with the rest.

Edit:
TrueOS 1712-STABLE seems broken, should be deleted from server.


#7

rumor has it they working on an update, to include the intel fixes


#8

I wish them success. I’m optimistic. Still loving TrueOS.

Thanks for quick response. You’re the best support, I’ve ever seen.


#9

Thanks bsdtester we’ll try that immediately. Regarding intel we saw i815 errors during install time (last stage) but thought it would not cause a problem, since we use different chipsets.


#10

However, the STABLE and UNSTABLE images seem to differ by 18 days only (12th December 2017 vs. 30th December 2017). There seems to be no February 12th, 2018 version available.

Index of /master/amd64/
…/
dist/ 12-Dec-2017 14:47 -
TrueOS-Desktop-17.12-x64-DVD.iso 12-Dec-2017 14:36 2898161664
TrueOS-Desktop-17.12-x64-DVD.iso.md5 12-Dec-2017 14:36 33
TrueOS-Desktop-17.12-x64-DVD.iso.sha256 12-Dec-2017 14:37 65
TrueOS-Desktop-17.12-x64-DVD.iso.sig 13-Dec-2017 14:45 543
TrueOS-Desktop-17.12-x64-DVD.iso.torrent 12-Dec-2017 14:37 221418
TrueOS-Desktop-17.12-x64-USB.img 12-Dec-2017 14:39 3120955904
TrueOS-Desktop-17.12-x64-USB.img.md5 12-Dec-2017 14:40 33
TrueOS-Desktop-17.12-x64-USB.img.sha256 12-Dec-2017 14:40 65
TrueOS-Desktop-17.12-x64-USB.img.sig 13-Dec-2017 14:46 543
TrueOS-Desktop-17.12-x64-USB.img.torrent 12-Dec-2017 14:40 238418
TrueOS-Server-17.12-x64-DVD.iso 12-Dec-2017 14:45 955801600
TrueOS-Server-17.12-x64-DVD.iso.md5 12-Dec-2017 14:45 33
TrueOS-Server-17.12-x64-DVD.iso.sha256 12-Dec-2017 14:46 65
TrueOS-Server-17.12-x64-DVD.iso.sig 13-Dec-2017 14:46 543
TrueOS-Server-17.12-x64-DVD.iso.torrent 12-Dec-2017 14:46 73234
TrueOS-Server-17.12-x64-USB.img 12-Dec-2017 14:46 1023451648
TrueOS-Server-17.12-x64-USB.img.md5 12-Dec-2017 14:47 33
TrueOS-Server-17.12-x64-USB.img.sha256 12-Dec-2017 14:47 65
TrueOS-Server-17.12-x64-USB.img.sig 13-Dec-2017 14:46 543
TrueOS-Server-17.12-x64-USB.img.torrent 12-Dec-2017 14:47 78395

Index of /unstable/amd64/
…/
dist/ 30-Dec-2017 21:29 -
TrueOS-Desktop-2017-12-30-UNSTABLE-x64-DVD.iso 30-Dec-2017 20:58 2805190656
TrueOS-Desktop-2017-12-30-UNSTABLE-x64-DVD.iso.md5 30-Dec-2017 20:58 33
TrueOS-Desktop-2017-12-30-UNSTABLE-x64-DVD.iso…> 30-Dec-2017 20:59 65
TrueOS-Desktop-2017-12-30-UNSTABLE-x64-DVD.iso.sig 03-Jan-2018 16:41 543
TrueOS-Desktop-2017-12-30-UNSTABLE-x64-DVD.iso…> 30-Dec-2017 20:59 214348
TrueOS-Desktop-2017-12-30-UNSTABLE-x64-USB.img 30-Dec-2017 21:01 3020579328
TrueOS-Desktop-2017-12-30-UNSTABLE-x64-USB.img.md5 30-Dec-2017 21:01 33
TrueOS-Desktop-2017-12-30-UNSTABLE-x64-USB.img…> 30-Dec-2017 21:01 65
TrueOS-Desktop-2017-12-30-UNSTABLE-x64-USB.img.sig 03-Jan-2018 16:42 543
TrueOS-Desktop-2017-12-30-UNSTABLE-x64-USB.img…> 30-Dec-2017 21:02 230788
TrueOS-Server-2017-12-30-UNSTABLE-x64-DVD.iso 30-Dec-2017 21:28 953518080
TrueOS-Server-2017-12-30-UNSTABLE-x64-DVD.iso.md5 30-Dec-2017 21:28 33
TrueOS-Server-2017-12-30-UNSTABLE-x64-DVD.iso.s…> 30-Dec-2017 21:28 65
TrueOS-Server-2017-12-30-UNSTABLE-x64-DVD.iso.sig 03-Jan-2018 16:42 543
TrueOS-Server-2017-12-30-UNSTABLE-x64-DVD.iso.t…> 30-Dec-2017 21:28 73084
TrueOS-Server-2017-12-30-UNSTABLE-x64-USB.img 30-Dec-2017 21:29 1020985856
TrueOS-Server-2017-12-30-UNSTABLE-x64-USB.img.md5 30-Dec-2017 21:29 33
TrueOS-Server-2017-12-30-UNSTABLE-x64-USB.img.s…> 30-Dec-2017 21:29 65
TrueOS-Server-2017-12-30-UNSTABLE-x64-USB.img.sig 03-Jan-2018 16:42 543
TrueOS-Server-2017-12-30-UNSTABLE-x64-USB.img.t…> 30-Dec-2017 21:29 78225


#11

I upgraded from STABLE (because freezing etc. etc.) in single user mode. Then upgrade worked over internet pc-updatemanager. But only the second time.
Result:
% zfs list | grep ’ /$'
tank/ROOT/12.0-CURRENT-up-20170901_183320 290K 1.28T 8.76G /
tank/ROOT/12.0-CURRENT-up-20180116_141657 372K 1.28T 9.74G /
tank/ROOT/12.0-CURRENT-up-20180118_232913 413K 1.28T 9.79G /
tank/ROOT/12.0-CURRENT-up-20180303_220220 24.0G 1.28T 8.90G /
tank/ROOT/initial 194K 1.28T 3.43G /

No new installation.

Edit:
My symptoms were the same as Yours.
Every package downloaded into staging area under /usr/… . But then: Nothing.


#12

O.K. We somehow fixed it, by listing all packages in the cache in /usr and then reinstalling and re-registering them.

Afterwards we forced an update of all local files with repository information, and started to install packages via pkg install one by one (about 30 packages with ~168 dependencies).

Later we again executed pc-updatemanger and realized the system now says its up to date. In the graphical manager we were prompted to restart, which we did.

After a restart the the installation continued, but the same error message (“Failed Updating!” and “Rolling back…”). This time we didn’t give a damn and rebooted into the last stable, which however has all the manual updates in it.

The fully booted system now looks like this:

That’s definitely not the right way to do things, but we had no alternative. By the way the pkg binary was up to date, bsdtester.

According to the screenshot we can now delete the latest, inactive boot environment (2018-03-07) and still have all updates in the active (NR) one.

Bottom line: something with booting into the new boot environment didn’t work, but we found nothing in the logs in /var/log so far. Any ideas where to look, devs? Could it be that changing the SWAP partition to /dev/ada0p3.eli and /dev/ada1p3.eli is the culprit? What are the requirements to boot into a new BE anyways?


#13

Regarding: ELI

Order is important.
First: Activate ELI.
Then: Activate SWAP

Else: Possibly, inconsistent amount of available space.

Maybe, I’ll give an example in the next couple of days. With real numbers and addresses of bytes and sectors, for others to check it for themselves.


#14

The space seems to be O.K. and we decrypt the partition during the boot process.

It seems we have to dive into the pc-updatemanger code and figure out under which conditions the system responds with “Failed Updating!” and “Rolling back…”.


#15

In https://github.com/trueos/pc-updatemanager/blob/master/rc-doupdate we found:

# We are doing a major upgrade!
TERM="cons25" ; export TERM
export PATH="${PATH}:/usr/local/bin:/usr/local/sbin"

if [ -e "/var/.pc-attempt1" ] ; then
   echo "The previous upgrade attempt failed!"
   echo "Rolling back to last known good BE..."
   oldBE=`cat /var/.lastGoodBE`
   /etc/beadm activate $oldBE
  /sbin/reboot
fi

A few lines later in the code…

# Set indicator that this is first attempt at upgrading
 touch /var/.pc-attempt1

This makes little sense to us. If its the first attempt you set the flag but you also check that flag (when?) and say the upgrade failed? Why?

In https://github.com/trueos/pc-updatemanager/blob/master/pc-updatemanager we found (seems less related to our bug, though):

#Doing a world upgrade, lets fetch FreeBSD packages
attempts=0
while :
do
attempts=$(expr $attempts + 1)
run_cmd_wtee "${PKG_CMD} ${PKG_CFLAG} ${PKG_FLAG} fetch -U -d -y -g FreeBSD-*" "${PKGUPGRADELOG}"
_err=$?
tail -n 1 ${PKGUPGRADELOG} | grep -q -e "failed checksum" -e "timed out" -e "not found" -e "No packages"
if [ $_err -ne 0 -o $? -eq 0 ]; then
  if [ $attempts -gt 3 ] ; then
    echo_log "*****" ${PKGUPGRADELOG}
    echo_log "Failed fetching: FreeBSD Base" >> ${PKGUPGRADELOG}
    exit_err "Failed fetching: FreeBSD Base"
  fi
else
  break
fi
done

Would one of the developers please explain this in greater detail and help us debug the problem? Thanks in advance!


#16
% cat /usr/local/share/trueos/pc-updatemanager/rc-doupdate | grep pc\-attempt1
if [ -e "/var/.pc-attempt1" ] ; then
touch /var/.pc-attempt1
rm /var/.pc-attempt1

First, check it:
Does it (still) exist? It shouldn’t.

Then:
Create it, before the action begins.

Lastly:
If the action didn’t abort prematurely/unsuccessfully ,remove it again.
(As hint for next run. Non-Presence == Success.)

Else keep it to show at start of next run that previous run didn’t complete (successfully).


That’s, how I read it. Looks good, so far. What’s wrong with it?


#17

You are right, bsdtester!

One of our engineers checked it and said the same (confirmed you). We figured out in the meantime, that the following was the critical part:

run_cmd_wtee "sh /.doPkgUp.sh" "$OLOG"
if [ $? -ne 0 ] ; then
  oldBE=`cat /var/.lastGoodBE`
  /etc/beadm activate $oldBE
  echo "Failed updating! Reverting to last good boot-environment..."
  sleep 10
  /sbin/reboot
fi

The shell script doPkgUp.sh was not executed with return code 0. We suspect the critical part is:

# Lock the TrueOS base package
pkg-static ${PKG_CFLAG} ${PKG_FLAG} lock -y ${SYSBASEORIGIN}
if [ $? -ne 0 ] ; then
  echo "FAILED LOCK: ${SYSBASEORIGIN}"
  sleep 10
  exit 1
fi

We saw such a message during the installation/upgrade process. We have to check the other logs. We realized there should be some logs like /usr/local/log/pc-updatemanager/upgrade.log and a lot more in /usr/local/log/pc-updatemanager. :slight_smile:

We hope we can find the root cause. Maybe it’s our fault and we messed up the test system…


#18

[-@-] ~% ls -slaph /usr/local/log/pc-updatemanager/

total 152
 2 drwxr-xr-x  2 root  wheel     8B Feb 13 14:48 ./
 1 drwxr-xr-x  3 root  wheel     3B Feb 13 15:27 ../
34 -rw-r--r--  1 root  wheel   103K Feb 13 14:48 current-pkg-list
 1 -rw-r--r--  1 root  wheel     0B Feb 13 14:48 failed-pkg-list
 2 -rw-r--r--  1 root  wheel   725B Feb 13 14:48 install-pkg-list
30 -rw-r--r--  1 root  wheel    86K Feb 13 14:48 previous-pkg-list
 1 -rw-r--r--  1 root  wheel     0B Feb 13 14:48 removed-pkg-list
84 -rw-r--r--  1 root  wheel   350K Feb 13 14:48 upgrade.log

Here we realized that graphics/phototonic phototonic-1.7.20_1.txz was uninstalled manually days ago. Still it is installed as a dependency?

[-@-] ~% sudo less /usr/local/log/pc-updatemanager/install-pkg-list

audio/pianobar pianobar-2016.06.02_4.txz
emulators/virtualbox-ose-additions virtualbox-ose-additions-5.2.0_1.txz
graphics/phototonic phototonic-1.7.20_1.txz
mail/trojita trojita-0.7_2.txz
misc/trueos-meta-hunspell trueos-meta-hunspell-20160701,1.txz
multimedia/libdvdcss libdvdcss-1.4.0_1.txz
multimedia/openh264 openh264-1.7.0,2.txz
multimedia/vlc vlc-2.2.6_2,4.txz
print/cups-pdf cups-pdf-2.6.1_4.txz
print/gutenprint gutenprint-5.2.13_1.txz
sysutils/fusefs-ntfs fusefs-ntfs-2017.3.23.txz
www/qupzilla-qt5 qupzilla-qt5-2.1.2_1.txz
x11-fonts/droid-fonts-ttf droid-fonts-ttf-20131024_3.txz
x11-fonts/noto-lite noto-lite-1.0.5.txz
x11-themes/cursor-jimmac-theme cursor-jimmac-theme-0.1_3.txz
x11/qterminal qterminal-0.8.0.txz

A No address record error in this logfile, besides No packages matched for pattern errors. Also pkg-1.10.1 [trueos-major] is reinstalled, but v1.10.5 is installed on our system now. Finally there is a No schema files found message and one tar: Error exit delayed from previous errors. message.

[-@-] ~% sudo less /usr/local/log/pc-updatemanager/upgrade.log

Fetching FreeBSD-librtld_db-lib32-development-12.0.s20171209183005.txz: ... done
Fetching FreeBSD-librtld_db-lib32-profile-12.0.s20171209183005.txz: ... done
pkg-static: http://pkg.cdn.trueos.org/master/amd64-base/FreeBSD-librtld_db-profile-    
12.0.s20171209183005.txz: No address record

Removing old packages... Please wait...
FreeBSD-acct-12.0.s20171209183005: already unlocked

[…]

zh-fcitx-configtool-0.4.8: already unlocked
zh-libchewing-0.5.1: already unlocked
zstd-1.3.2: already unlocked
zu-hunspell-20110323_1: already unlocked
No packages matched for pattern 'multimedia/cuse4bsd-kmod'
No packages matched for pattern 'graphics/drm-next-kmod'
No packages matched for pattern 'sysutils/freecolor'
No packages matched for pattern 'graphics/gpu-firmware-kmod'
No packages matched for pattern 'devel/libstatgrab'
No packages matched for pattern 'sysutils/lsof'
No packages matched for pattern 'security/lynis'
No packages matched for pattern 'editors/nano'
No packages matched for pattern 'sysutils/pefs-kmod'
No packages matched for pattern 'sysutils/personacrypt'
No packages matched for pattern 'misc/trueos-desktop'

Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 533 packages 
(of 0 packages in the universe):

Installed packages to be REMOVED:
        ORBit2-2.14.19_2
        aalib-1.4.r5_11

[…]

[75/533] Deinstalling cups-pdf-2.6.1_4...
You may need to manually remove /usr/local/etc/cups/cups-pdf.conf if it is no longer 
needed.
[75/533] Deleting files for cups-pdf-2.6.1_4: ....... done

[…]

The following 1 package(s) will be affected (of 0 checked):

Installed packages to be REINSTALLED:
        pkg-1.10.1 [trueos-major]

Number of packages to be reinstalled: 1
[1/1] Reinstalling pkg-1.10.1...

[…]

|   |   | `-- Extracting glib-2.50.2_7,1: .......... done
No schema files found: doing nothing.
|   |   | `-- Installing glproto-1.4.17...

[…]

You installed BSDstats: script reporting statistics about your machine

To disable monthly reporting, add this line to /etc/periodic.conf:
        monthly_statistics_enable="NO"

[…]

Locking trueos-desktop-201712111405
Extracting distribution files...
x ./etc/rc.shutdown
x ./etc/auto_master
x ./etc/rc
x ./etc/devd.conf
x ./etc/rc.devd
x ./etc/defaults/rc.conf
tar: /etc/autofs: Not found in archive
tar: Error exit delayed from previous errors.
* rc-update: abi already installed in runlevel `boot'; skipping

[-@-] ~% less /pkg-add.log

|   | `-- Extracting qt4-inputmethods-4.8.7_2: .... done
|   `-- Extracting zh-fcitx-4.2.9.1_4: .......... done
ldconfig: warning: yes: No such file or directory
`-- Extracting fcitx-m17n-0.2.3: .......... done

[-@-] ~% ls -slapht /usr/local/log/pc-updatemanager/failed-pkg-list

1 -rw-r--r--  1 root  wheel     0B Feb 13 14:48 /usr/local/log/pc-updatemanager/failed-pkg-list

Could one of the errors above have doPkgUp.sh function run_cmd_wtee() made return something other than zero, thus failing run_cmd_wtee “sh /.doPkgUp.sh” “$OLOG” in rc-doupdate?


TRON-DELTA.ORG – TrueOS PC-Updatemanager Security Review 2018
#19

Beanpole135, was the last post helpful? Were you or the other devs able to locate the problem or make the update scripts more resilient?


#20

I believe it was.
I sent Kris the link to this thread (he is the one that maintains pc-updatemanager) and he got everything up and running again. In the future though, it would be more helpful to create a bug ticket on our GitHub repository so that we can properly track the issue and ensure it gets resolved.