Migrating from NextCloud to Infinite Scale ocis

There are too many cloud variants and use cases out there to have a migration path for all at hand, but let’s see what we can start with: There is the famous Swiss army knife for clouds called rclone available.

The awesome rclone tool makes it easy to migrate data from one installation to another on a user per user base. A very good first step is to install it using command postmaster net/rclone.

This article explains by the example of Nextcloud how you would migrate your data from an running NC to Infinite Scale.

To be able to address the clouds in rclone, you need to configure so called remotes. It is nothing else than a shortcut for the combination of

• which kind of cloud are you talking to
• the URL
• the username
• the password, if one is set

You need to add a configuration for both the source cloud (Nextcloud) and the target (Infinite Scale). As both talk WebDAV, the rclone manual page is accurate to follow.

For both, use an URL in the form of https://my.host.local/remote.php/webdav.

Once that is finished, the command rclone config show should give output similar to this:

[:~/] ± rclone config show

[NCoC]
type = webdav
url = https://nc.local/remote.php/webdav
vendor = nextcloud
user = wilma
pass = zfdsaiewrafdskfjdasfxdasffdafdsafas

[ocis]
type = webdav
url = https://infinitescale.local/remote.php/webdav
vendor = owncloud
user = wilma
pass = cdsfasrefdsadaGkxTXjksfpqQFI5nQawqs

Now, for example the directories on the Nextcloud root can be checked with rclone lsd NCoC:/.

To migrate the data, rclone provides the command copy. It transfers data from one remote to the other. Use the following command example to transfer the entire cloud data from Nextcloud to Infinite Scale:

rclone  copy NCoC:/ ocis:/ --no-check-certificate -P

The –no-check-certificate can and should be skipped if your clouds have proper certificates. The -P however, provides you with interesting statistics about the copy progress. Of course you can use move command instead of copy in rclone.
Once you are finished, this might be the result:

[:~/] $ rclone copy NCoC:/ ocis:/ --no-check-certificate -P
Transferred:        1.228 GiB / 1.228 GiB, 100%, 10.170 MiB/s, ETA 0s
Transferred:          411 / 411, 100%
Elapsed time:      2m19.3s

Note that while testing this, occasionally the Nextcloud was returning a 404 not found for some files. While the reason for that was not completely clear, it does not matter, because the rclone command can be repeated. It is clever enough to only copy what has changed!

Basic Authentication is disabled by default in oCIS because of security considerations. In order to make the following Rclone commands work the oCIS administrator needs to enable Basic Authentication e.g. by setting the environment variable PROXY_ENABLE_BASIC_AUTH to true.

That's why we need a shell script to set some needed and important environment variables. Let name it ocis.sh and put it into /home/ocis directory:

#!/usr/bin/env csh

cd /home/ocis/
setenv HOME /home/ocis
setenv OCIS_URL https://owncloud.ostreff.info
setenv PROXY_HTTP_ADDR 0.0.0.0:9200
setenv PROXY_ENABLE_BASIC_AUTH true
setenv OCIS_INSECURE true
setenv OCIS_LOG_COLOR false
setenv OCIS_LOG_PRETTY false
setenv DEMO_USERS false                        # do not create demo users
setenv OCIS_LDAP_INSECURE true

./ocis server

Building the ocis is really simple on FreeBSD. I have cloned the git repository from https://github.com/owncloud/ocis/ and then use following command to build the source:

gmake generate && \
gmake -C ocis clean && \ 
gmake -C ocis build && \
mv ocis/bin/ocis /home/ocis/ocis

Here is my initial variant of rc.d for FreeBSD:

#!/bin/sh

# PROVIDE: ocis
# REQUIRE: DAEMON
# KEYWORD: shutdown

. /etc/rc.subr

name="ocis"
rcvar=ocis_enable

: ${ocis_enable:="NO"}
: ${ocis_user:="ocis"}
: ${ocis_group:="ocis"}
: ${ocis_tag:="ocis"}

# read configuration and set defaults
start_precmd="ocis_precmd"
stop_postcmd="ocis_shutdown"

load_rc_config ${name}

ocis_precmd()
{
    install -d -o ${ocis_user} -g ${ocis_group} /var/run/${name}/
    install -d -o ${ocis_user} -g ${ocis_group} /var/log/${name}/
}

ocis_shutdown()
{
  if [ -e "${pidfile}" ]; then
    echo "Stopping supervising daemon."
    kill -s TERM `cat ${pidfile}`
  fi
}

pidfile="/var/run/${name}/${name}.pid"
procname="/home/${name}/${name}.sh"
command="/usr/sbin/daemon"
command_args="-S -p ${pidfile} -o /var/log/${name}/${name}.log -T ${ocis_tag} -t ${ocis_tag} ${procname}"

run_rc_command "$1"