Upgrade on Linux

info

Please read carefully before starting the upgrade process.

Introduction

Depending on whether there have been changes to the DBmarlin repository schema between the currently installed version and target version, the upgrade may be very fast or could take some time.

  • Upgrading between versions where the DBmarlin schema has not changed is a quick process and just requires the new tar.gz file to be extracted over the top of the old installation while all DBmarlin processes are stopped and then running configure.sh.
  • Upgrading between versions where the DBmarlin schema has has changed, will take longer and will require a period of downtime while the data is migrated to the new schema.

Please follow the pre-requistes first to find out which case you are dealing with. The upgrade steps will be the same in either case but the time taken to complete will be different.

Pre-requisites

Follow the pre-requisites below to check the repository schema version for the version you already have and the version you are looking to install.

Backup existing installation

It is a good idea to have a backup of the current installation to rollback to in the event that the upgrade is unsucessful. A directory copy or filesystem snapshot while the DBmarlin processess have been stopped would give you something to restore back if needed.

Download the latest version

Download the dbmarlin-Linux-x64-[VERSION].tar.gz file from https://www.dbmarlin.com/download

Check the current schema version

You can see which version you have installed and the version of the repository schema it is using by looking at the version.txt file in the root of the DBmarlin intallation directory. For example:

cat version.txt
ui-version=1.0.0
schema-version=1.3

Check the new schema version to be installed

Extract the tar.gz of the newer version into a temporary location and do the same check of the version.txt file to check the schema-version. If the schema-version is the same as the one you already have, then no data migration is required and the upgrade will be quick. If the schema-version is newer than the one you already have, then data migration is required which could take some time. Additionally, you must check you have enough disk space before proceeding.

Checking disk space

If the schema-version has changed then you should make sure that you have enough disk space on the volume where DBmarlin is installed to accommodate double the amount of data, plus 10 percent headroom, for a short period of time while migration completes. You can use du to get the disk usage for the DBmarlin directory and df to get the disk space available on the file system which DBmarlin is installed on. For example:

cd /opt/dbmarlin
du -hs .
6.1G .
df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/centos-root 97G 11G 87G 11% /

In the example above DBmarlin is using 6.1GB so would required an extra 6.71GB (6.1GB +10%) free space to complete the upgrade. The filesystem has 87GB available so we are ok to proceed.

info

Once the data migration has completed and all data has been copied to the new schema, the old schema will be automatically dropped freeing up the extra disk space.

Upgrade steps

Stop all DBmarlin server components

Stop all DBmarlin processeses and make sure they are all stoppped.

./stop.sh
./status.sh
nginx (Not running)
tomcat (Not running)
postgres (Not running)

Extract the tar.gz file

Extract the newer tar.gz over the top of the current install. For example if DBmarlin is installed in /opt/dbmarlin then run:

cd /opt
tar -xzvf dbmarlin-Linux-x64-[VERSION].tar.gz

Configure the installation

After extracting the tar.gz file you will find configure.sh which needs to be run first before you can start up the server. This configures ports and other settings.

cd /opt/dbmarlin
./configure.sh
Press Enter then space bar to scroll
...
Do you accept the agreement Y/N ? y
Done.
Set port for Nginx (current 9090): 9090
Set port for Tomcat (current 9080): 9080
Set port for PostgreSQL (current 9070): 9070
See https://docs.dbmarlin.com/docs/Getting-Started/hardware-requirements for Profile sizes
1) XSmall
2) Small
3) Medium
4) Large
5) Xlarge
Choose the profile [1-4]: 2
Small profile selected
Recommended RAM is 4GB. You have 4G
Continue Y/N ? y
...

The configure.sh will check if this is a first time installation or if there is an existing schema and whether that schema is already the correct version or if a new schema needs to be created.

Scenario 1 - No schema update necessary

If the schema version is the same you will see a message like this indicating that no schema change required and the upgrade is now complete. You can run start.sh to restart all processes.

Detected the following schema versions:
v1_3
You have the latest schema already. No schema update necessary.
All done!
Next run ./start.sh to startup the DBmarlin services

Scenario 2 - New schema and data migration required

If a new schema is needed, you will see a message like this. This means a new schema is needed. The data migration will start in the background once you run start.sh.

Detected the following schema versions:
v1_2
Checking for enough disk space. You need double + 10 percent during a schema upgrade.
DBmarlin is using: 6343440 K
Filesystem free is: 91144588 K
Space required is: 13955568 K
Freespace is OK for schema upgrade.
...
Data will copy from the old 1_2 schema to the new 1_3 schema in the background once you run start.sh. Please be patient. It could take some time.
All done!
Next run ./start.sh to startup the DBmarlin services
info

Data migration could take several minutes or even several hours for large databases. During this time you will not be able to start any sensors until the migration has completed.

Restart all DBmarlin server components

./start.sh
./status.sh
nginx (Running)
tomcat (Running)
postgres (Running)

Checking the upgrade progress

You can tail the latest Tomcat localhost log file to see where the upgrade has got up to. The biggest tables activity and statistic are the ones which will take longest time. Please be patient.

cd /opt/dbmarlin/tomcat/logs
tail -f localhost.2020-11-30.log | grep --line-buffered Data-migrator
30-Nov-2020 13:23:48.714 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate data from v1_2 to v1_3
30-Nov-2020 13:23:48.741 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: stop sensors 3, 17, 18, 8, 14, 5, 7, 4, 12, 1, 2, 15, 9, 16, 19, 6, 10, 13, 11, 20
30-Nov-2020 13:23:48.762 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: restart sequences
30-Nov-2020 13:23:48.763 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: restart sequence for licence_id at 4
30-Nov-2020 13:23:48.806 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: restart sequence for agent_id at 2
30-Nov-2020 13:23:48.812 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: restart sequence for datasource_id at 21
30-Nov-2020 13:23:48.816 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: restart sequence for event_type_id at 5
30-Nov-2020 13:23:48.826 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: restart sequence for event_id at 2
30-Nov-2020 13:23:48.829 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate tables
30-Nov-2020 13:23:48.829 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table licence
30-Nov-2020 13:23:48.926 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table integration
30-Nov-2020 13:23:48.950 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table agent
30-Nov-2020 13:23:48.977 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table datasource
30-Nov-2020 13:23:49.018 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table host
30-Nov-2020 13:23:49.047 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table change
30-Nov-2020 13:23:49.304 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table custom_event_type
30-Nov-2020 13:23:49.336 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table custom_event
30-Nov-2020 13:23:49.372 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table sql
30-Nov-2020 13:23:49.539 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table sql_plan
30-Nov-2020 13:24:25.958 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table activity_1hour
30-Nov-2020 13:24:54.550 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table activity
30-Nov-2020 13:34:53.698 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table statistic_1hour
30-Nov-2020 13:34:55.000 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: migrate table statistic
30-Nov-2020 13:47:55.460 INFO [pool-5-thread-1] org.apache.catalina.core.ApplicationContext.log Data-migrator: data migration took 1447031ms

Once complete you will see a log message showing the total time for the migration E.g. Data-migrator: data migration took 1447031ms. You can now go back to the DBmarlin UI in your browser and see that the sensors have are back to their previous state prior to the upgrade; if they were stopped, then they will need to be manually restarted, otherwise if they were started then they should now have restarted and new data should start being collected.

info

If you try to start/stop/edit a sensor which the upgrade is in progress you will be prevented from doing so and the sensor status will report 'Stopped - upgrading'.

Refresh the UI

It is recommended that following an upgrade that you do a hard refresh in your browser to make sure that you have the latest versions of all files loaded. shift + cmd + R on MacOS or Ctrl + F5 on Windows.

Troubleshooting

Check the log file

You can tail the latest Tomcat localhost log file to see where the upgrade has got up to. For example:

cd /opt/dbmarlin/tomcat/logs
vi localhost.2020-11-30.log

You can search the log for Data-migrator to find lines related to the data migration.

Check in PostgreSQL if the migration is still running

You can query pg_stat_activity to see the active sessions and whether the migration process is still running or is blocked on something such as a lock.

cd /opt/dbmarlin
echo "select * from pg_stat_activity where state <> 'idle';" | ./postgresql/bin/psql

Check the output to see if the migration is still prgressing or if it is blocked on something.

Rolling back to the previous version

If the data migration got part way through before failing then you may have both the old and new schema versions. You should clean up the new schema that was created so that you can retry the upgrade again. We provide a drop_schema.sql script for this.

List the schema versions in the database

cd /opt/dbmarlin
echo "\dn" | ./postgresql/bin/psql | grep -o 'v[[:digit:]]*_[[:digit:]]' | sort

Run the drop_schema.sql script BEFORE you restore the tar.gz from the previous version. This is so the newer schema is dropped and not the older one you want to roll back to.

cd /opt/dbmarlin
cat ./postgresql/sql/drop_schema.sql | ./postgresql/bin/psql

To go back to the old version you can now run ./stop.sh to stop all processes and then extract the tar.gz file from the old version back over the top of the current installation and then run ./start.sh

Retrying a failed upgrade

To try an upgrade again you should first rollback to the previous version as descibed above. Then run through the upgrade steps again.

Contact support

Reach out to support via one of the contact options listed on https://www.dbmarlin.com/support