ElementsServices.DataSync package

Submodules

ElementsServices.DataSync.ConnectionConfiguration module

The ConnectionConfiguration Module

class ElementsServices.DataSync.ConnectionConfiguration.ConnectionConfiguration(filename=None)[source]

Bases: object

The connection configuration mainly holds: * the host type and URL, * the user name and password, * the overwriting policy.

overwritingAllowed()[source]

Check whether overwriting a local file is allowed.

parseConfigurationFile(filename)[source]

Parse the connection configuration file.

parseHost(name)[source]

Parse the name of a data hosting solution (case-insensitive).

parseOverwritingPolicy(policy)[source]

Parse the Overwriting Policy

class ElementsServices.DataSync.ConnectionConfiguration.DataHost[source]

Bases: enum.Enum

The test data hosting solution.

IRODS = 'iRODS'
WEBDAV = 'WebDAV'
class ElementsServices.DataSync.ConnectionConfiguration.OverwritingPolicy[source]

Bases: enum.IntEnum

The overwriting policy if the local file already exists.

ABORT = 0
OVERWRITE = 1

ElementsServices.DataSync.DataSyncUtils module

ElementsServices.DataSync.DataSyncUtils.concatenatePaths(chunks)[source]

Concatenate path chunks into a single path.

ElementsServices.DataSync.DataSyncUtils.createLocalDirOf(local_file)[source]

Create the parent directory for a local file.

ElementsServices.DataSync.DataSyncUtils.dataSyncConfFilePath(filename)[source]

Get the path of a configuration file for the data synchronization tool.

ElementsServices.DataSync.DataSyncUtils.environmentVariable(name)[source]

Get the value of an environment variable or ‘’ if it does not exist.

ElementsServices.DataSync.DataSyncUtils.localDirExists(local_dir)[source]

Check whether a local directory exists.

ElementsServices.DataSync.DataSyncUtils.localWorkspacePrefix()[source]

Get the prefix of the local workspace.

ElementsServices.DataSync.DataSyncUtils.runCommandAndCaptureOutErr(cmd)[source]

Execute a command and return its output and error messages.

ElementsServices.DataSync.DataSynchronizer module

class ElementsServices.DataSync.DataSynchronizer.DataSynchronizer(connection, dependencies)[source]

Bases: object

A data synchronizer class is able to download test data from a host.

This is the abstract class to be extended for each data host.

createDownloadCommand(distant_file, local_file)[source]

Create the command to download a file.

downloadAllFiles()[source]

Download all the test files.

downloadOneFile(distant_file, local_file)[source]

Download a given test file.

fileAlreadyExists(local_file)[source]

Check whether a file to be downloaded already exists locally.

fileShouldBeWritten(local_file)[source]

Check whether a file should be locally written.

hasBeenDownloaded(distant_file, local_file)[source]

Check whether a given test file has been downloaded, i.e., exists and is not empty.

exception ElementsServices.DataSync.DataSynchronizer.DownloadFailed(distant_file, local_file)[source]

Bases: Exception

An exception raised when downloading fails.

ElementsServices.DataSync.DataSynchronizerMaker module

ElementsServices.DataSync.DataSynchronizerMaker.createSynchronizer(connection_config, dependency_config)[source]

Create the appropriate data synchronizer as specified in a given connection configuration.

ElementsServices.DataSync.DependencyConfiguration module

DependencyConfiguration Module

class ElementsServices.DataSync.DependencyConfiguration.DependencyConfiguration(distant_root, local_root, config_file=None)[source]

Bases: object

The dependency configurations holds, for each test file to be retrieved: * the distant source path, * the local destination path.

getAliasSeparator()[source]

Get the separator between the filename and its alias.

getDependencyCount()[source]

Get the number of files to be downloaded.

getDistantPathOf(local_file)[source]

Get the distant source path associated to a local destination path.

getDistantPaths()[source]

Get the list of all distant source paths.

getFileMap()[source]

Get the local-distant file map.

getLocalPaths()[source]

Get the list of all local destination paths.

lineHasAlias(line)[source]

Check whether a line of the dependency configuration file contains an alias.

parseConfigurationFile(filename)[source]

Parse a dependency configuration file.

parseConfigurationLine(line)[source]

Parse a line of the dependency configuration file.

parseLineWithAlias(line)[source]

Parse a line of the dependency configuration file which contains an alias.

parseLineWithoutAlias(line)[source]

Parse a line of the dependency configuration file which does not contain an alias.

ElementsServices.DataSync.IrodsSynchronizer module

class ElementsServices.DataSync.IrodsSynchronizer.IrodsSynchronizer(connection, dependencies)[source]

Bases: ElementsServices.DataSync.DataSynchronizer.DataSynchronizer

A data synchronizer for iRods hosts.

createDownloadCommand(distant_file, local_file)[source]

Create the command to download a file.

ElementsServices.DataSync.IrodsSynchronizer.irodsIsInstalled()[source]

Check whether an iRODS client is installed.

ElementsServices.DataSync.WebdavSynchronizer module

class ElementsServices.DataSync.WebdavSynchronizer.WebdavSynchronizer(connection, dependencies)[source]

Bases: ElementsServices.DataSync.DataSynchronizer.DataSynchronizer

A data synchronizer for WebDAV hosts.

createDownloadCommand(distant_file, local_file)[source]

Create the command to download a file.

ElementsServices.DataSync.WebdavSynchronizer.webdavIsInstalled()[source]

Check whether an iRODS client is installed.

Module contents

Main DataSync Package

class ElementsServices.DataSync.DataSync(connectionFile, dependencyFile)[source]

Bases: object

A class to download test data from a data repository prior to the unit test run.

Refer to the corresponding Redmine project for more details.

Parameters:
  • connectionFile – Path to the connection configuration file relative to the configuration directory.
  • dependencyFile – Path to the dependency configuration file relative to the configuration directory.

See also

https://euclid.roe.ac.uk/projects/testdata/wiki

absolutePath(relativePath)[source]

Get the absolute path to a local test file which has been downloaded.

The absolute path is the concatenation of:

  • a system-defined prefix,
  • the local workspace,
  • the path of the file relative to the local workspace.

On LODEEN, the prefix is empty. On CODEEN, it is the job workspace which the user do not know. It can be set by the user through the $WORKSPACE environment variable.

Warning

This function must be used to access any data downloaded by the DataSync tool.

download()[source]

Download the test data.

downloadWithFallback(connectionFile)[source]

Download the test data and provide a fallback host in case the primary host fails.

Parameters:connectionFile – Path to the connection configuration file of the fallback host relative to the configuration directory.